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Abstract 

Maintenance  data  collection  is  an  integral  part  of  flightline  aircraft  maintenance. 
Historically,  this  data  was  input  via  traditional  keyboard  data  entry  methods  at  a 
computer  terminal.  These  terminals  are  typically  located  in  the  aircraft  maintenance  unit 
(AMU)  facility,  away  from  where  the  actual  maintenance  is  being  performed.  In  contrast 
to  the  traditional  approach,  the  Point-of-Maintenance  system  (POMX)  seeks  to  reduce  the 
data  entry  burden  while  increasing  data  accuracy  through  the  use  of  E-Tools  such  as 
ruggedized  laptop  computers  and  handheld  portable  maintenance  aids  (PMAs).  POMX 
enables  data  entry  at  the  aircraft  or  other  maintenance  location  via  wireless  local  area 
network  or  batch  storage,  and  seeks  to  capture  data  as  the  maintenance  is  performed. 

This  research  analyzes  the  impact  of  a  POMX  system  on  maintenance  data  error 
rates.  This  research  takes  a  careful  look  at  the  implementation  of  POMX  at  Randolph 
AFB  to  enable  current  designers  and  system  engineers  to  gain  insight  into  what  to  expect 
as  the  next  generation  of  POMX  comes  on-line.  Initial  results  indicate  no  significant 
improvement  in  data  quality  and  no  reduction  in  the  number  of  data  errors  recorded  with 
POMX  systems.  Follow-up  interviews  with  POMX  users  and  experts  revealed  that  the 
Air  Force  still  has  a  number  of  managerial,  technical  and  organizational  constraints 
which  must  be  overcome  before  a  POMX  system  can  add  to  the  effectiveness  of  Air 
Force  maintenance  operations. 
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IMPROVING  MAINTENANCE  DATA  COLLECTION  VIA 


POINT-OF-MAINTENANCE  (POMX)  IMPLEMENTATION 


I.  Introduction 


Background 

Maintenance  data  collection  (MDC)  is  an  integral  part  of  flightline  aircraft  maintenance. 
Until  recent  years,  the  Core  Automated  Maintenance  System  (CAMS)  was  the  Air  Force’s 
primary  MDC  database  with  the  exception  of  Air  Mobility  Command  (AMC),  which  used  a 
slightly  different  system  called  G08 1  -  CAMS  for  Mobility.  CAMS  and  G08 1  are  presently 
being  phased  out  in  favor  of  an  enterprise-wide  solution  known  as  the  Integrated  Maintenance 
Data  System  (IMDS).  IMDS  is  part  of  the  larger  logistics  enterprise  architecture  (LogEA)  which 
serves  as  “the  operational  and  systems  architecture  that  will  support  the  execution  of  the  eLog2 1 
campaign”  (LogEA  CONOPS,  2004).  eLog21  is  a  senior-leader  initiative  to  reshape  logistics  for 
the  21st  century  to  better  support  the  network-centric  warfare  efforts  of  the  Department  of 
Defense  (DOD). 

MDC  has  historically  been  a  tedious  and  time-consuming  task  resulting  in  marginal  data 
accuracy.  Inputs  under  the  CAMS  and  G08 1  systems  occur  via  traditional  keyboard  data  entry 
methods  at  a  computer  terminal.  These  tenninals  are  typically  located  in  the  aircraft 
maintenance  unit  (AMU)  facility,  away  from  where  the  actual  maintenance  is  being  performed. 
Inputs  are  manually  reviewed  and  errors  are  identified  and  corrected  by  a  member  of  the 
organizational  level  data  integrity  team  (DIT).  In  contrast  to  the  manual  approach,  the  Point-of- 
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Maintenance  system  (POMX)  seeks  to  reduce  the  data  entry  burden  while  increasing  data 
accuracy  through  the  use  of  E-Tools  such  as  handheld  portable  maintenance  aids  (PMAs). 

Efforts  to  streamline  the  data  collection  process  are  not  a  new  endeavor.  An  obvious 
example  is  the  pervasive  use  of  point-of-sale  technology  in  the  retail  industry.  Besides 
enhancing  the  consumer  checkout  process  by  shortening  customer  wait  time,  the  bar-code 
scanners  and  UPC  codes  in  use  today  have  resulted  in  automated  and  comprehensive  data 
collection  for  inventory  management  purposes  while  effectively  eliminating  keypunch  entry 
errors. 

POMX  is  an  alternative  front-end  method  of  interfacing  with  IMDS  which  is  represented 
in  Figure  1  by  the  CAMS  database.  It  is  part  of  the  eLog21  initiative  sponsored  by  Headquarters 
Air  Force/Logistics,  Installations  &  Mission  Support/Maintenance  Management  branch 
(AF/A4MM)  and  managed  by  the  Automatic  Identification  Technology  (AIT)  Program 
Management  Office  (PMO)  at  Wright-Patterson  AFB.  The  POMX  architecture  includes  a 
wireless  local  area  network  (LAN)  that  serves  as  the  backbone  of  the  system,  ruggedized 
handheld  tenninals  (HHTs)  for  use  by  maintenance  technicians,  a  dedicated  POMX  server  which 
receives  the  data  and  synchronizes  it  with  IMDS,  laptop  computers  for  in-vehicle  use  by 
flightline  expediters,  desktop  computer  interfaces  for  system  administration  and  analysis,  and 
handheld  radios  that  interface  with  a  speech  conversion  box  connected  to  a  voice  recognition 
application  resident  on  the  POMX  server.  Figure  1  provides  a  graphical  representation  of  the 
architecture  and  hardware  components. 


2 


POMX  Server 


wvwwvwt 

■M/WVw — 


Flightline 
FM  Radio 


System 
FM  Radio 


Speech 

Conversion 

Box 


A/WWWWV 

w\/\/y\/vv — 

FIPS  140  Compliant 


WVWVWWA/ 

— wy/yyvvv — 

FIPS  140  Compliant 


Rugged  ized 
Laptops 


Desktop 

Computer 


CAMS 

Database 


Support  Multiple  Configurations: 
Windows,  Handheld,  Laptop,  Desktop 


Figure  1.  POMX  Architecture  (Klarer,  2005) 

POMX  enables  data  entry  at  the  aircraft  or  other  maintenance  locations,  and  seeks  to 
capture  data  as  the  maintenance  is  performed.  This  is  in  contrast  to  the  legacy  approach  of 
computer  terminal  keyboard  entry  in  a  location  removed  from  the  flightline  environment.  The 
legacy  method  typically  resulted  in  MDC  being  delayed  until  the  technician  was  able  to  return  to 
the  aircraft  maintenance  unit  (AMU)  often  resulting  in  incomplete  and  inaccurate  data. 

POMX  also  provides  an  intelligent  interface  via  the  POMX  server  to  guard  against 
incorrect  input  by  performing  error  checking  routines  as  information  is  entered.  Certain  logical 
combinations  of  input  data  are  required  based  on  a  series  of  decision  rules  and  categorical 
assignments.  This  seeks  to  create  a  fail-safe  environment  and  eliminates  both  inadvertent  and 
intentional  erroneous  inputs  and  their  underlying  causes  as  identified  in  two  different  studies  by 
Folmar  (1986)  and  Determan  (1991). 
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Problem  Statement 


The  current  state  of  the  POMX  initiative  is  focused  on  designing  and  implementing  the 
necessary  architecture  to  enable  the  Automatic  Identification  Technology  (AIT)  requirements  of 
eLog21  and  fulfill  the  point-of-maintenance  requirements  of  the  IMDS  system.  Existing  POMX 
contracts  involve  detailed  functional  requirements  rather  than  broad  perfonnance  objectives.  It 
is  not  clear  to  what  extent  POMX  will  impact  data  accuracy  once  these  functional  requirements 
are  met.  The  purpose  of  this  research  is  to  determine  whether  the  POMX  implementation  at 
Randolph  AFB,  Texas  led  to  an  increase  in  the  accuracy  of  maintenance  data  when  compared  to 
the  legacy  method  of  MDC. 

Research  Question 

Does  implementation  of  Point-of-Maintenance  (POMX)  at  Randolph  AFB,  Texas  provide 
significant  improvement  to  flightline  maintenance  data  accuracy  over  the  legacy  method  of 
inputting  data  via  IMDS  computer  tenninals  located  at  the  aircraft  maintenance  unit  (AMU)? 
Two  investigative  questions  are  considered  during  the  course  of  this  research  to  aid  in  answering 
this  question. 

Investigative  Questions 

IQ  1 .  Does  the  POMX  system  at  Randolph  AFB  provide  a  significant  reduction  in  MDC 
error  rates  when  compared  to  legacy  systems? 

IQ2.  Do  aircraft  maintenance  units  at  Randolph  AFB  experience  a  reduction  in  specific 
types  of  errors  as  a  result  of  using  POMX  when  compared  to  legacy  systems? 

Research  Objective 

The  overarching  objective  of  this  research  is  to  analyze  the  impact  of  a  POMX  system  on 
data  accuracy.  Accurate  data  is  an  essential  goal  of  the  Air  Force’s  eFog21  initiative.  In  an 
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effort  to  ensure  current  efforts  are  headed  in  the  right  direction,  this  study  seeks  to  validate  the 
idea  of  POMX  as  a  valuable  tool  to  ensure  the  Air  Force  technician  is  capturing  good 
maintenance  data.  Additionally,  skilled  Air  Force  aircraft  maintenance  technicians  are  an 
underutilized  resource  when  they  are  performing  tedious  proofreading  tasks  instead  of 
performing  aircraft  maintenance  tasks  for  which  they  were  trained.  The  potential  exists  for 
POMX  to  eventually  eradicate  the  need  for  error  checking  and  reporting.  This  research  takes  a 
careful  look  at  the  implementation  of  POMX  at  Randolph  AFB  to  enable  current  designers  and 
system  engineers  to  gain  insight  into  what  to  expect  as  the  next  generation  of  POMX  comes  on¬ 
line. 

Scope  and  Limitations 

This  research  is  designed  to  support  the  POMX  effort  currently  sponsored  by 
Headquarters  Air  Force/Logistics,  Installations  &  Mission  Support/Maintenance  Management 
branch  (AF/A4MM)  and  managed  by  the  Air  Force  Automated  Information  Technology 
Program  Management  Office.  Due  to  delays  in  user  evaluations,  data  were  not  available  to 
perform  analysis  on  the  most  current  POMX  system  being  tested  by  the  Air  Force  at  Hurlburt 
Field,  FL. 

It  was  initially  thought  that  several  other  bases  were  actively  using  various  forms  of 
POMX  systems  to  perform  flightline  maintenance.  As  this  research  effort  continued  to  mature,  it 
became  evident  that  the  bases  originally  thought  to  be  actively  using  POMX  were  suffering  from 
implementation  issues  and  were  not  using  MDC  systems  in  a  manner  consistent  with  the  POMX 
data  collection  methodology  use.  These  bases  included  Nellis  AFB,  Nevada,  Langley  AFB, 
Virginia,  and  Moody  AFB,  Georgia.  The  only  base  that  was  confirmed  as  currently  using 
POMX  technology  was  Randolph  AFB  which  implemented  the  system  during  June- July  2003. 
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Though  Randolph  is  a  training  base,  it  is  believed  that  the  results  of  this  study  provide  valuable 
information  regarding  the  future  potential  of  the  both  the  current  POMX  venture  sponsored  by 
AF/A4MM  and  of  the  various  other  wireless  efforts  taking  place  in  the  Air  Force.  Compared  to 
the  architecture  depicted  in  Figure  1,  the  Randolph  AFB  system  in  Figure  2  is  both  less  complex 
and  less  capable  at  this  point  in  time.  It  consists  of  the  IMDS/CAMS  database  which  is  accessed 
by  desktop  computers  via  a  hardwired  local  area  network  (LAN)  or  by  ruggedized  laptops  via  a 
wireless  LAN.  Maintainers  access  the  system  on  the  flightline  by  using  the  laptops.  There  are 
no  handheld  terminals  and  no  dedicated  POMX  server.  Additionally,  it  does  not  use  voice 
recognition  technology.  Although  this  hardware  configuration  contains  less  functionality  than 
the  system  this  research  originally  hoped  to  evaluate,  it  provides  the  point-of-maintenance 
capability  called  for  in  the  IMDS  requirements.  For  this  reason,  it  is  a  valid  representation  of  the 
POMX  concept. 


Desktop 

Computer 


Ruggedized 

Laptops 


WLAN 


or 

Docking 

Station 


CAMS 

Database 


Figure  2.  Randolph  AFB  POMX  Architecture  (Adapted  from  Klarer,  2005) 


Methodology 

This  research  implements  an  ex  post  facto  experimental  design  using  quantitative 
analysis  of  MDC  error  data  reported  by  the  maintenance  analysis  organization  at  Randolph  AFB, 
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TX.  This  data  consists  of  two  data  sets  comprised  of  monthly  error  rates.  There  are  24  data 
points  collected  prior  to  the  implementation  of  POMX  and  24  collected  after  POMX 

Analysis  consists  of  a  series  of  individual  tests  of  hypothesis  to  determine  if  significant 
differences  exist  between  the  mean  error  rates  of  the  populations.  Due  to  the  small  sample  sizes 
(n  <  30),  the  t-statistic  is  used  to  compute  the  appropriate  rejection  regions.  Individual  results  are 
reported  as  each  hypothesis  is  evaluated.  The  findings  are  also  considered  in  light  of  the  current 
POMX  program’s  objectives,  goals,  and  expectations  in  an  effort  to  identify  potential 
shortcomings  or  pitfalls. 

Summary 

The  current  POMX  initiative  is  only  a  small  piece  of  the  Air  Force’s  future  logistics 
environment.  As  the  requirements  for  accurate  information  have  developed  to  support  an 
integrated  logistics  system,  the  accuracy  of  the  maintenance  data  affects  not  only  the  demand 
placed  on  the  supply  chain,  but  also  weapon  system  sustainment  efforts.  The  ability  to  capture 
information  as  tasks  are  being  perfonned  is  believed  to  be  a  factor  in  ensuring  accurate  data. 

This  research  seeks  to  add  credibility  to  this  claim  and  provide  further  basis  for  upcoming 
decisions  on  POMX  implementation  and  expected  outcomes. 

The  next  chapter  reviews  literature  relevant  to  the  POMX  concept.  This  review  provides 
the  frame  of  reference  which  serves  as  the  basis  for  this  research.  The  framework  establishes  the 
importance  of  this  study  and  assists  the  reader  in  understanding  how  the  study  was  conceived. 
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II.  Literature  Review 


Chapter  Overview 

Increased  data  accuracy  attributable  to  the  Point-of-Maintenance  (POMX)  system  is  the 
focus  of  this  research,  and  such  a  discussion  has  a  natural  progression.  Beginning  with  the 
importance  of  data  accuracy,  this  literature  review  explores  both  present  and  future  Air  Force 
requirements.  Current  guidance  regarding  aircraft  maintenance  management  describes  legacy 
systems  and  practices,  while  documentation  regarding  the  Air  Force’s  logistics  transformation 
campaign  (eLog21)  provides  the  foundation  that  drives  the  POMX  initiative  and  demands  its 
successful  implementation.  Nearly  error  free  data  are  absolutely  essential  to  the  current  logistics 
transformation  efforts  that  enable  network-centric  warfare.  Next,  this  review  explores  some  of 
the  reasons  for  maintenance  data  collection  errors  as  reported  in  the  findings  of  two  Air  Force 
Institute  of  Technology  (AFIT)  studies.  Following  that  discussion  is  the  description  of  the 
POMX  concept  as  reported  in  contractual  documents,  news  releases,  and  test  reports.  This 
includes  isolated  system  implementation  efforts  as  well  as  the  current  program  sponsored  by 
Headquarters  Air  Force.  Finally,  the  literature  review  concludes  by  capturing  how  the  POMX 
initiative  integrates  into  the  future  Air  Force  logistics  system  to  meet  the  data  accuracy 
requirements  mentioned  in  the  first  part  of  this  discussion. 

The  Importance  of  Data  Accuracy 

Aircraft  maintenance  is  an  extremely  regimented  and  detailed  endeavor.  This  is  a 
necessity  not  only  to  ensure  successful  mission  accomplishment,  but  more  so  because  the  lives  of 
aircrew  and  passengers  rest  in  the  hands  of  the  technicians,  managers,  and  senior  leaders  who 
make  it  their  job  to  ensure  all  aircraft  and  equipment  under  their  charge  are  mission  capable. 


8 


Fundamental  to  the  maintenance  mission  are  the  Air  Force  core  values  of  “integrity  first,  service 
before  self,  and  excellence  in  all  we  do.” 

Integrity  in  the  aircraft  maintenance  world  addresses  not  only  the  individuals  perfonning 

their  duties,  but  also  inanimate  aspects  such  as  equipment  and  infonnation.  Good  infonnation 

enables  good  decision  making  while  incorrect  infonnation  can  lead  decision  makers  to  fall  well 

short  of  their  goals.  Air  Force  Instruction  (AFI)  21-101  describes  integrity  as  follows: 

Maintenance  discipline  involves  integrity  in  all  aspects  of  the  maintenance  process.  It  is 
the  responsibility  of  all  maintenance  personnel  to  comply  with  all  written  guidance  to 
ensure  all  required  repairs,  inspections,  and  documentation  are  completed  in  a  safe, 
timely,  and  effective  manner  (AFI  21-101,  2004). 

The  instruction  also  stresses  the  importance  of  data  accuracy  in  the  following  discussion 

regarding  maintenance  management  metrics: 

Leaders,  supervisors  and  technicians  must  have  accurate  and  reliable  information  to  make 
decisions.  Maintenance  management  metrics  are  a  crucial  form  of  information  used  by 
maintenance  leaders  to  improve  the  performance  of  maintenance  organizations, 
equipment,  and  people  when  compared  to  established  goals  and  standards.  Metrics  must 
be  accurate  and  useful  for  decision-making  (AFI  21-101,  2004). 

Useful  and  accurate  metrics  serve  as  a  baseline  necessary  to  support  an  environment  of 

continuous  organizational  improvement.  Today’s  global  environment  and  rapidly  changing 

battle-space  demand  a  sustained  focus  on  continuous  improvement  to  ensure  the  Air  Force  is 

able  to  outthink  and  outperform  its  enemies.  The  Air  Force  has  detennined  that  error  free  data  is 

of  enough  importance  to  make  Maintenance  Group  Commanders  (MXG/CC)  responsible  to: 

Ensure  aircraft  maintenance  data  is  accurate  by  establishing  and  supporting  a  data 
integrity  team  (DIT).  Ensure  members  assigned  to  the  DIT  are  suitably  qualified  and 
provided  sufficient  time  to  accurately  assess  the  data.  Ensure  each  aircraft  maintenance 
work  center  performs  a  review  of  all  documentation  entered  into  CAMS/G081  daily  IAW 
TO  00-20-2,  Maintenance  Data  Documentation  (AFI  21-101,  2004). 

The  data  integrity  team  (DIT)  consists  of  representatives  from  each  maintenance 

squadron  in  addition  to  representatives  from  the  Maintenance  Data  Systems  Analysis  (MDSA) 
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section  who  work  directly  for  the  MXG/CC  and  lead  the  DIT  effort.  More  commonly  there  are 

DIT  members  from  each  flight  within  a  squadron,  and  those  individual  are  charged  with  ensuring 

the  accuracy  of  the  data  within  their  flight.  To  do  this,  they  enlist  the  help  of  workcenter 

representatives  within  their  flight.  A  typical  flightline  unit’s  representatives  might  include  a 

crew  chief,  electrician,  hydraulic  technician,  jet  engine  mechanic,  avionics  technician,  and 

possibly  an  egress  technician.  These  technically  qualified  individuals  are  charged  with  the  daily 

review  of  all  job  data  documentation  (JDD)  input  into  CAMS  or  G081  in  order  to  find,  correct, 

and  document  errors.  This  is  a  tedious  and  time  consuming  task  that  is  hardly  foolproof.  Few 

maintainers  relish  the  idea  of  paperwork  when  they  could  be  turning  wrenches,  so  careful 

selection  is  important  when  choosing  DIT  members.  While  every  Air  Force  member  should 

strive  to  maintain  the  core  value  of  “integrity  first”,  the  most  conscientious  individuals  should  be 

chosen  to  perform  these  daily  workcenter  reviews  to  find  and  correct  documentation  errors. 

For  the  purposes  of  this  research,  the  Air  Force  Instruction  is  used  as  the  baseline  criteria  upon 

which  the  premise  of  the  DIT  is  established.  The  instruction  reads  as  follows: 

The  purposes  of  the  Data  Integrity  Team  include:  (1)  ensuring  the  unit  has  complete  and 
accurate  data  in  the  MIS  and  aircraft  forms  (to  include  all  inputs  made  by  staff  agencies), 
(2)  identifying  and  quantifying  problems  within  the  unit  preventing  complete  and 
accurate  documentation,  (3)  identifying  and  correcting  the  root  causes  for  poor  data 
integrity,  and  (4)  educating  the  unit  on  the  critical  need  for  data  integrity.  The  DIT/DIG 
teams  are  established  to  evaluate/isolate/eliminate  documentation  problems  in 
CAMS/G08 1  (AFI  21-101,  2004). 

The  future  of  maintenance  data  collection  (MDC)  in  the  Air  Force  is  designed  to  depart 
from  outdated  practices  of  manual  keyboard  entry  at  a  computer  terminal  located  in  the  aircraft 
maintenance  unit.  Legacy  processes  consist  of  multiple  logistics  systems  that  are  slow  and 
fragmented.  These  systems  often  have  difficulty  talking  to  one  another,  and  in  many  instances 
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there  is  no  interface  at  all.  In  a  presentation  by  Mr.  Grover  Dunn  on  the  subject  of  Air  Force 
Logistics  Transformation,  he  points  out  the  following: 

Implementation  of  the  new  logistics  architecture  (LogEA)  focuses  on  three  key  elements. 

1 .  Establishing  a  customer- focused  enterprise- wide  perspective 

2.  Developing  a  lean,  fully  integrated  logistics  enterprise 

3.  Data  exploitation  and  predictive  logistics  awareness  (Dunn,  2005). 

This  enterprise-wide  integrated  logistics  system  will  provide  the  Air  Force  a 
comprehensive  capability  that  links  all  of  the  major  logistics  functions  and  their  information 
systems.  Two  of  the  six  architecture  design  principles  outlined  in  the  LogEA  Concept  of 
Operations  (CONOPS)  focus  on  maintaining  the  data.  First,  the  design  calls  for  a  single  data 
model  which  will  serve  as  the  enterprise  data  warehouse  for  all  logistics  data.  This  serves  to 
eliminate  redundant  and  conflicting  data  while  greatly  increasing  the  speed  at  which  decision 
makers  can  understand  relevant  information.  It  also  emphasizes  data  integrity  and  quality  by 
dictating  that  “every  data  element  has  only  single  point  of  entry”,  and  that  “all  edits  for  a  data 
element,  and  for  all  data  in  the  transaction,  are  performed  at  time  of  entry”  (LogEA  CONOPS, 
2004). 

POMX  is  the  data  entryway  into  the  Integrated  Maintenance  Data  System  (IMDS)  which 
is  one  of  the  components  of  the  Expeditionary  Combat  Support  System  (ECSS),  the  enterprise 
resource  planning  (ERP)  system  the  Air  Force  has  selected  to  meet  the  operating  and  systems 
architectures  defined  in  the  LogEA  (LogEA  Fact  Sheet,  2004).  The  LogEA  also  includes  a 
detailed  transformation  plan  that  serves  as  a  roadmap  to  fulfill  the  eLog2 1  campaign  goals  of 
providing  a  fully  integrated  logistics  enterprise  by  incorporating  all  logistics  processes  into  a 
common  framework  (LogEA  CONOPS,  2005).  ECSS  is  the  actual  software  that  accomplishes 
this  by  integrating  “...over  500  legacy  infonnation  technology  systems  with  a  Commercial-of- 
the-Shelf  (COTS)  IT  suite”  (ECSS  Fact  Sheet,  2004).  ECSS  will  be  readily  accessible  via  the 
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Air  Force  Portal’s  Global  Combat  Support  System  (GCSS-AF),  and  will  provide  comprehensive 
actionable  information  coupled  with  decision  support  tools  to  aid  users  at  all  levels  of  the 
decision  making  chain. 

Today,  systems  engineers,  program  managers,  and  industry  consultants  are  taking  great 
care  to  ensure  data  is  entered  accurately  the  first  time.  This  involves  developing  highly 
intelligent  systems  that  make  use  of  error  checking  routines  to  quickly  alert  an  individual  if  they 
enter  erroneous  data.  Much  of  the  effort  focuses  on  how  the  system  will  capture  data  and  is 
being  designed  to  automate  the  process  to  the  maximum  extent  possible.  Mr.  Grover  Dunn 
addresses  this  as  part  of  the  logistics  transfonnation  vector  of  “Accurate  timely  and  secure 
integrated  enterprise  data”  (Dunn,  2005).  While  the  POMX  functionality  has  great  potential  to 
ease  the  maintainer’s  burden,  it  is  important  to  remember  that  it  is  not  intended  to  simply  be  an 
update  to  the  CAMS  interface. 

In  summary,  it  is  clear  that  the  Air  Force  is  dedicated  to  obtaining  accurate  data  through 
manual  error  checking  by  dedicated  personnel  or  through  the  intelligent  design  of  systems.  In 
order  to  make  the  transition  from  legacy  to  future  state  data  collection,  designers  must  have  an 
understanding  of  the  types  and  causes  of  data  errors.  This  is  discussed  in  the  upcoming  section. 

Data  Entry  Errors 

There  are  multiple  reasons  why  data  entry  errors  may  occur.  In  attempting  to  understand 
the  cause  of  inaccurate  information,  it  is  necessary  to  begin  with  a  framework  that  concedes  that 
human  factors  play  a  major  role  and  will  never  be  a  variable  which  can  be  fully  controlled.  Two 
studies  on  data  entry  are  looked  at  in  this  review.  The  first  piece  of  research  discusses  the  fact 
that  some  personnel  appear  to  input  errors  intentionally,  while  the  second  study  attempts  to 
capture  the  inherent  causes  of  data  input  errors.  Both  works  are  deemed  to  be  appropriate 
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representations  of  legacy  maintenance  data  collection  systems  to  include  CAMS  and  G081  as 
well  as  Air  Force  Technical  Order  (AFTO)  349  paper  forms  which  are  still  used  when  the 
electronic  systems  go  offline. 

In  the  first  study,  Folmar  looked  at  intentional  input  errors  in  1986  by  conducting  a 
survey  of  430  maintenance  personnel  from  Strategic  Air  Command  (SAC)  and  434  from  Tactical 
Air  Command  (TAC).  The  survey  return  rate  was  a  rather  high  58  percent,  resulting  in  249 
responses  from  TAC  and  256  from  SAC.  Folmar’s  research  was  conducted  prior  to  fleet-wide 
implementation  of  the  CAMS  system.  MDC  at  that  time  consisted  of  AFTO  Form  349s  to 
document  maintenance  job  data.  Folmar  found  that  inaccurate  data  was  intentionally  input  into 
the  MDC  system  approximately  10  percent  of  the  time,  and  the  reason  most  commonly  listed  for 
this  was  “pressure  from  supervisors  and  managers  to  account  for  100  percent  of  their  man-hour 
availability”  (Folmar,  1986).  While  this  was  the  most  common  reason  cited  for  intentional 
errors,  this  result  ranked  fourth  when  it  came  to  the  causes  of  errors  in  general.  The  most 
common  reason  noted  for  data  errors  was  that  “personnel  failed  to  take  the  necessary  time  to  fill 
out  the  paperwork  right”  (Folmar,  1986).  This  was  followed  by  lack  of  proper  training  and 
understanding  of  the  complex  maintenance  data  collection  system. 

In  addition,  Folmar’s  research  captured  individual  perceptions  of  the  value  of  the  MDC 
information  that  they  were  expected  to  enter.  Maintenance  personnel  were  not  convinced  of  the 
necessity  of  the  maintenance  data  they  were  entering.  Although  they  survey  found  there  was  a 
slight  consensus  that  the  information  provided  useful  infonnation  for  base-level  decision  makers, 
personnel  were  ambivalent  regarding  the  timeliness  of  the  information  for  management  purposes. 
Folmar  sums  up  the  question  of  individual’s  perceptions  as  follows: 

Maintenance  personnel  feel  that  the  data  collected  is  important  for  base-level  managers 

but  the  current  MDC  system  is  much  less  than  ideal.  They  see  the  data  base  as  being  full 
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of  inaccuracies  and  invalid  inputs.  They  feel  the  current  system  is  flawed,  slow  and 
tedious.  They  would  like  to  see  changes  made  to  make  the  system  quicker  and  more  user 
oriented.  Many  hoped  that  the  CAMS  data  collection  system  would  be  implemented  as 
soon  as  possible  (Folrnar,  1986). 

In  the  second  study,  Detennan  conducted  similar  research  5  years  later  once  CAMS  had 
become  the  Air  Force’s  primary  means  of  MDC.  436  personnel  were  surveyed  from  both  TAC 
and  SAC,  with  respondents  numbering  180  and  191  respectively  for  a  42.5  percent  return  rate. 
Determan’s  results  were  similar  to  Folmar’s  in  that  approximately  10  percent  of  the  time 
personnel  felt  the  errors  were  intentional.  Intentional  causes  included  pressure  to  falsify 
information,  lack  of  perceived  benefit  from  accurate  information,  and  difficulty  in  entering  the 
information  into  CAMS  (Detennan,  1991).  In  open-ended  responses,  additional  reasons  reported 
included  the  following: 

CAMS  won’t  accept  the  correct  information  so  erroneous  infonnation  is  entered  to  clear 
the  job  (15  responses),  personnel  are  lazy  and  not  motivated  to  enter  the  correct 
information  (7  responses),  and  personnel  are  not  adequately  trained  on  CAMS  (8 
responses)  (Detennan,  1991). 

Accidental  errors  were  reported  to  be  caused  by  keystroke  errors,  lack  of  training  in  using  the 
technical  order  (T.O.)  system,  and  lack  of  training  in  using  the  CAMS  system.  This  was  further 
explained  by  comments  on  how  difficult  it  was  to  find  the  correct  codes  in  the  T.O.,  and  that 
navigating  the  multiple  screens  required  to  document  job  data  in  CAMS  was  confusing 
(Detennan,  1991).  When  asked  “what  is  the  single  most  beneficial  action  we  could  take  to 
reduce  or  eliminate  data  entry  enors  in  CAMS?”,  responses  were  to  provide  CAMS  training, 
make  the  system  more  user  friendly,  and  provide  on-line  help  functionality  within  the  system. 
Finally,  there  was  a  perceived  improvement  in  data  accuracy  at  the  supervisor  and  manager  level, 
but  not  at  the  worker  level  (Detennan,  1991). 
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While  CAMS  seems  to  be  an  improvement  over  paper  documentation,  maintenance  data 
collection  remains  a  tedious  and  time-consuming  task.  Documentation  frequently  still  takes 
place  away  from  the  aircraft  and  well  after  the  maintenance  is  performed.  It  is  often  the  current 
practice  to  complete  several  jobs  before  returning  to  the  maintenance  unit  to  document  the  job 
information,  as  leaving  the  flightline  after  every  job  to  “go  do  CAMS”  wastes  precious  time  that 
can  be  utilized  for  aircraft  maintenance.  It  is  thought  that  more  accurate  data  could  be  captured 
at  the  aircraft  as  soon  as  the  maintenance  action  took  place,  only  in  recent  years  has  it  become 
technologically  feasible.  The  next  section  describes  the  evolution  of  the  Point-of-Maintenance 
(POMX)  concept  as  a  way  to  capture  CAMS  and  G08 1  data  with  minimal  inconvenience  to 
maintenance  personnel. 

POMX  Development 

The  POMX  concept  originated  as  a  requirement  for  the  Air  Force’s  new  Integrated 
Maintenance  Data  System  (IMDS)  to  have  the  “ability  to  capture  maintenance  data  at  point  of 
origin”  (Gober,  2003).  When  dealing  with  the  acquisition  of  a  new  information  system,  it  often 
takes  time  for  the  requirements  to  fully  mature.  This  has  never  been  more  evident  than  in 
today’s  world  of  rapidly  advancing  technology  and  spiral  development  programs  based  on 
incremental  requirements  and  the  fielding  of  systems  in  multiple  blocks  of  progressive 
capability.  IMDS  is  no  different,  and  it  has  taken  time  for  developers  to  fully  understand  what 
the  POMX  capabilities  should  be.  This  section  reviews  several  wireless  local  area  network 
(LAN)  implementations  that  have  fallen  under  the  broad  umbrella  of  POMX  as  well  as  the 
official  Air  Force  sponsored  program  that  is  sponsored  by  AF/A4MM  and  run  by  the  AIT 
Program  Management  Office  at  Wright-Patterson  AFB,  Ohio. 
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A  2001  study  by  the  Logistics  Management  Institute  (LMI)  explored  the  following 
objectives  while  capturing  the  current  state  of  Portable  Maintenance  Aid  (PMA)  use: 

•  Assess  current  and  emerging  PMA  concepts,  equipment  and  programs  for  DoD 
weapon  systems. 

•  Characterize  potential  PMA  benefits. 

•  Identify  implementation  desirability,  issues,  and  challenges  (Bapst  et  ah,  2001). 
The  team  accomplished  this  goal  by  examining  33  applications  of  PMA  use  in  both  military  and 
civilian  settings.  It  was  noted  that  the  Department  of  Defense  (DOD)  could  obtain  the  following 
benefits  by  pursuing  PMA  use. 

•  Give  maintainers  on-the-job  access  to  electronic  technical  infonnation, 
maintenance  documentation,  and  parts  availability  data. 

•  Provide  a  capability  for  maintainers  to  enter  repair  data  from  job  sites  in  real  time 
in  order  to  provide  instantaneous  visibility  to  multiple  users  regarding 
maintenance  actions  and  equipment  status. 

•  Enable  on-the-job  access  to  detailed  technical  data  and  remote  engineering 
support  via  automated  information  technology  (AIT)  and  telemaintenance. 

•  Allow  direct  communications  with  weapon  systems  in  order  to  facilitate 
troubleshooting,  particularly  for  complex  systems  that  do  not  have  embedded 
diagnostic  and  prognostic  functionality  (Adapted  from  Bapst  et  ah,  2001). 

Examples  of  the  various  types  of  PMAs  evaluated  are  depicted  in  Figure  3. 
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Figure  3.  Sample  PMA  Hardware  (Bapst  et  al.,  2001). 

Some  of  the  devices  in  Figure  2.1  are  older  devices  that  were  only  used  as  the  Air  Force 
began  exploring  the  advantages  of  collecting  data  at  the  point-of-maintenance,  however  the 
evolution  of  this  technology  has  closely  followed  the  advances  of  personal  and  laptop  computers 
and  portable  data  assistants  (PDAs).  As  these  devices  matured,  they  became  smaller,  lighter,  and 
more  capable.  One  of  the  earlier  test  locations  for  these  devices  was  Randolph  AFB,  TX,  where 
these  ruggedized  laptops  were  used  to  test  a  wireless  LAN  in  October  2002  in  an  effort  to  bring 
POMX  to  the  Air  Education  and  Training  Command  (AETC)  architecture.  Ruggedized  laptop 
computers  provided  by  Itronix  were  used  to  connect  to  the  wireless  LAN  and  gain  access  to 
CAMS  and  SBSS.  While  slow  log-on  times  and  the  short  battery  life  of  the  laptops  were 
problematic,  AETC  News  Service  quoted  one  avionics  technician  who  noted  “The  wireless  local 
area  network  laptop  is  convenient  and  saves  time”  (AETC  News  Service,  2003).  Overall,  the  test 
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results  were  promising,  and  the  base  officially  adopted  the  new  system  in  July  2003.  A  similar 
effort  took  place  shortly  thereafter  at  Nellis  AFB,  NV  where  the  57th  Aircraft  Maintenance 
Squadron  implemented  the  Itronix  system  as  the  first  Air  Combat  Command  (ACC)  base  to  go 
wireless  (Network  World,  2004).  Since  these  early  tests,  many  other  bases  in  various 
commands  have  adopted  various  versions  of  POMX  and  wireless  LANs  with  both  laptop  and 
handheld  terminal  data  entry  based  on  the  most  current  technology  and  funding  available  at  the 
time  of  fielding. 

As  POMX  began  to  emerge  as  an  idea  garnering  Headquarters  Air  Force  level  attention, 
independent  testing  of  version  1.0  was  conducted  in  a  collaborative  effort  between  the  Air  Force 
Research  Laboratory  (AFRL),  the  University  of  Dayton  Research  Institute  Human  Factors 
Group,  and  NCI  Information  Systems,  Inc.  The  testing  that  took  place  from  4-7  November, 
2002  at  Hurlburt  Field,  FL  and  was  designed  to  answer  the  following  questions: 

1.  For  opening  jobs,  ordering  parts,  and  closing  jobs  on  the  flightline,  how  does  mode  of 
operation  (real-time  with  POMX,  batch  with  POMX,  or  baseline  without  POMX) 
affect  performance? 

2.  Which  mode  of  operation  do  users  prefer? 

3.  Is  information  presented  on  POMX  device  readable  and  usable  (Gorman,  et  ah, 

2003)? 

The  hardware  used  during  the  test  included  the  Intermec  710  handheld  device  with  POMX 
software,  extra  batteries  and  charger  for  the  device,  and  a  CAMS  terminal  based  on  the  legacy 
“green  screen”  type  display  (Gorman,  et  al.,  2003). 

Test  results  were  mixed  as  the  software  did  not  function  entirely  as  expected.  The 
researchers  found  that  the  only  function  in  question  1  they  were  able  to  test  was  the  ability  to 
open  jobs,  and  the  real-time  functionality  was  hampered  by  RF  connectivity  difficulties. 
Nevertheless,  the  team  was  able  to  test  the  POMX  system  using  the  batch  mode  and  compare 
those  input  times  to  the  CAMS  input  times.  Results  were  promising  as  the  benefits  from  being 
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able  to  capture  the  data  at  the  point-of-maintenance  resulted  in  significant  improvement  over  the 
CAMS  data  entry  method.  Even  when  not  considering  travel  time  to  the  AMU  building  where 
the  CAMS  tenninal  was  located,  the  POMX  method  of  opening  a  job  showed  that  subjects  took 
an  average  of  5  minutes  41  seconds  to  complete  the  task  compared  to  8  minutes  13  seconds  using 
CAMS.  When  travel  time  to  the  CAMS  terminal  was  considered,  the  CAMS  task  took  an 


average  of  10  minutes  46  seconds  (Gorman,  2003).  The  report  also  noted  relative  acceptance  of 
the  POMX  handheld  system  based  on  subjective  user  feedback.  Most  users  felt  the  handheld 
device  was  more  user  friendly  than  the  CAMS  environment  when  considered  separately  from  the 
frustrations  experienced  with  the  software  and  wireless  LAN. 

This  early  testing  eventually  led  to  continued  systems  engineering  and  the  POMX 
acquisition  program  that  is  currently  in  development.  Since  version  1 .0,  the  system  has  followed 
the  following  progression  as  reported  in  a  13  April  2005  Status  Update  by  the  AIT  PMO: 


_ Table  1.  POMX  Progression _ 

Version _ Status/Issues _ Testing 

2.0/2. 1  Expanded  Year  1  capability  Nov-03 

2.2  Updated  Synchronization  Process  Mar-05 

3.0  Addressing  software  speed  and  synchronization  issues.  Mid-2005 

Software  problems  have  been  an  ongoing  concern.  AIT  PMO 
will  not  accept  the  system  version  until  the  initial  handheld 
terminal  synchronization  is  2  minutes  or  less,  and  subsequent 
updates  20  seconds  or  less. 


3.0  Qualification  Testing  &  Evaluation 

3.0  User  evaluations  at  Hurlburt  Field,  Florida 

3.1  In  development 


Scheduled  for 
late  March-06 
Scheduled  for 
Aug-06 
Unknown 


(Adapted  from  Klarer,  2005). 

Currently,  POMX  version  3.0  enables  the  following  processes  as  listed  in  the  version  3.0 
Interface  Design  Description  (IDD): 
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Voice  Process:  Provides  an  alternative  functionality  of  updating  the  POMX  Database 
Server  with  maintenance  transactions  voiced  in  over  the  FM  radio  to  be  verified  via  the 
handheld  tenninal  (HHT).  Table  2  details  a  list  of  the  voice  commands  currently  being 
developed  to  use  this  technology  (CDO  Technologies,  Inc.,  2004). 


Table  2.  Voice  Commands 


Create  Job 

Add  Profile 

Create  Unscheduled  Job 

Reopen  Event 

Schedule  Job 

Order  Part 

Create  WCE  (Work  Center  Event) 

Micap 

Schedule  WCE 

Back  Order 

Schedule  Cann  (Cannibalization  Action) 

Status  Update 

Schedule  TCTO  (Time  Compliance  Technical  Order) 

(CDO  Technologies,  Inc.,  2004) 

Database  Synchronization  Process:  Provides  functionality  of  synchronization  (Bi- 
Directional  Replication)  of  the  POMX  Database  Server  (Oracle  9i  Database)  with  the 
Handheld  Databases  (Sybase  UltraLite).  The  maintenance  events  that  were  recorded  at 
the  Point  of  Maintenance  needs  to  be  sent  to  CAMS  for  processing  and  the  maintenance 
transaction  results  processed  at  CAMS  needs  to  be  sent  back  to  the  handheld  device. 
POMX  Database  Synchronization  Process  provides  the  functionality  of  sending  and 
receiving  the  transactions  between  the  POMX  Server  Database  and  the  handheld 
databases. 

Web  Application:  Provides  functionality  to  review  and  update  the  data  required  for 
support  of  the  POMX  processing. 

POMX  CAMS  Processor  (PCP):  Retrieves  from  the  queue  on  the  POMX  Database 
Service  the  transactions  to  be  processed  through  to  CAMS. 

Query  Language  Program  (QLP)  Importer  Process:  Provides  functionality  of 
updating  the  POMX  Database  Server  with  the  up  to  date  job  information  and  look  up 
table  information  from  the  Core  Automated  Maintenance  System  (CAMS). 

RF  Messaging  Process:  RF  Messaging  Process  provides  functionality  of  immediately 
sending  a  Voiced  in  Record  to  the  appropriate  Handheld  Terminal  (HHT).  When  a 
POMX  user  voices  in  a  transaction  to  be  run  through  CAMS,  they  will  immediately 
receive  a  message  on  the  HHT  stating  that  a  new  voice  record  has  been  received.  They 
will  then  be  able  to  verily  and  send  the  transaction  to  the  POMX  database  to  be  processed 
into  CAMS  (CDO  Technologies,  Inc.,  2004). 

Also  detailed  in  the  version  3.0  IDD  are  sixteen  transaction  sets  that  POMX  will  allow  a 

maintainer  to  perform.  These  transactions  encompass  the  same  functionality  previously 
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available  only  through  CAMS,  and  will  afford  maintainers  a  distinct  advantage  in  minimizing  the 
potential  to  input  bad  data. 


Table  3.  Transaction  Sets 


Create  Maintenance  Event 

Query  Supply 

Schedule  Event  (various  types) 

TCTO  Kit  Order  Event 

Reschedule  Maintenance  Event/Workcenter  Event 

Corrective  Action  Event 

Reschedule  Unscheduled  Discrepancy 

Change  Event/Workcenter  Event  Narrative 

Reopen  Maintenance  Event 

Status  Update 

Create  Workcenter  Event 

Defer  Job 

Order  Part  Event 

Configuration  Management 

Bench  Stock  Order 

Error  Results  Sets 

(CDO  Technologies,  Inc.,  2004) 


Summary 

This  chapter  began  by  making  the  case  for  the  importance  of  aircraft  maintenance  data 
accuracy  as  a  tool  for  decision  making.  It  also  made  the  connection  between  POMX  and  the 
larger  logistics  transfonnation  efforts  taking  place  under  the  eLog21  campaign  as  structured  by 
the  Logistics  Enterprise  Architecture  and  enabled  by  the  Expeditionary  Combat  Support  System. 
This  was  followed  by  an  exploration  of  some  earlier  studies  that  quantified  causes  of  data  entry 
errors  under  older  maintenance  data  collection  (MDC)  systems.  The  chapter  concluded  with  a 
description  of  POMX  attributes,  capabilities,  and  expected  benefits,  as  well  challenges  faced  by 
users  during  testing  of  the  system  tools.  This  study  proceeds  in  the  next  chapter  with  a 
discussion  of  the  research  methodology.  This  encompasses  the  details  of  how  the  study  will 
address  the  investigative  questions  which  in-turn  are  used  to  answer  the  research  question. 
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III.  Methodology 


Chapter  Overview 

Having  established  the  background  for  the  POMX  effort  in  the  previous  chapter,  this 
section  provides  the  methods  and  rationale  used  to  answer  the  research  question.  This  research 
uses  quantitative  data  analysis  to  accomplish  this  goal.  Following  a  review  of  the  research 
question,  this  chapter  discusses  the  specific  techniques  used  to  answer  the  investigative 
questions.  The  questions  are  discussed  in  terms  of  underlying  assumptions,  data  organization, 
and  tests  of  hypotheses.  In  order  to  build  the  framework  of  understanding  that  will  answer  the 
investigative  questions,  it  is  helpful  to  review  these  questions  in  the  context  established  by  the 
literature  review  and  overall  research  problem. 

Research  Question  Review  and  Structure 

As  previously  noted,  the  purpose  of  this  research  is  to  determine  whether  POMX  leads  to 
increased  data  accuracy  over  the  legacy  method  of  inputting  data  into  the  IMDS  system  via  a 
stationary  computer  tenninal  located  in  the  AMU  at  Randolph  AFB.  To  answer  this,  the  two 
investigative  questions  (IQs)  are  used  to  conduct  a  quantitative  comparison  of  maintenance  data 
error  rates  between  POMX  and  legacy  systems.  In  conducting  this  type  of  analysis,  these 
questions  capture  the  state  of  the  current  system  in  use  and  shed  light  on  any  changes  in  data 
accuracy  attributable  to  the  POMX  concept.  This  infonnation  should  be  considered  by  the 
current  POMX  acquisition  program  to  demonstrate  advances  in  data  quality  achieved  to  date 
through  POMX  use.  The  investigative  questions  are  considered  in  more  detail  in  the  section  that 
follows. 

IQ  1 .  Does  the  POMX  system  at  Randolph  AFB,  TX  provide  a  significant  reduction  in 

MDC  errors  when  compared  to  legacy  systems? 
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IQ2.  Are  specific  types  of  MDC  errors  at  Randolph  AFB,  TX  reduced  by  using  POMX 
when  compared  to  legacy  systems? 

To  answer  these  questions,  a  post  hoc  experimental  design  is  used  in  which  a  series  of 
hypotheses  are  proposed  and  tested.  These  are  further  explained  in  a  later  discussion  under  the 
heading  of  hypothesis  testing. 

Assumptions 

It  is  noteworthy  that  Randolph  AFB  is  unique  in  several  ways  compared  to  the  other 
mentioned  bases.  First,  Randolph  AFB  utilizes  contract  aircraft  maintenance,  therefore  the 
maintenance  being  performed  and  documented  is  not  accomplished  by  active  duty  Air  Force 
personnel.  Many  contract  maintainers  are  former  active  duty  members  who  are  familiar  with  the 
airframes  and  information  systems  they’re  using,  and  those  who  are  not  prior  service  members 
have  undergone  similar  extensive  aircraft  maintenance  training  and  certification  via  other 
programs.  Second,  Randolph  AFB  is  an  Air  Education  and  Training  Command  (AETC)  training 
base  and  typically  operates  somewhat  differently  from  bases  in  commands  such  as  Air  Combat 
Command  (ACC)  and  Air  Mobility  Command  (AMC).  For  the  purposes  of  this  research,  the 
personnel  are  believed  to  have  common  skills  and  training  for  entering  data  into  POMX,  CAMS, 
or  G081,  and  there  is  no  substantial  difference  among  maintainers  at  various  bases,  whether 
active  duty  or  civilian  contract  personnel.  It  is  therefore  assumed  that  POMX  data  entry  has  a 
similar  potential  to  improve  data  accuracy  regardless  of  base  or  organization.  It  is  also  believed 
that  lessons  learned  from  this  study  should  be  applicable  to  other  POMX  efforts  in  the  Air  Force. 

During  the  course  of  this  research,  it  was  confirmed  that  Randolph  AFB  attempts  to  use 
POMX  as  much  as  possible,  but  there  are  still  problems  related  to  the  wireless  connectivity  that 
sometimes  prevent  its  use.  These  connectivity  problems  were  also  cited  by  Nellis,  Langley,  and 
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Moody  when  questioned  about  why  they  were  not  actively  attempting  to  use  their  systems.  It 
appears  that  Randolph  has  enjoyed  a  relative  sustained  level  of  success  for  their  POMX  system 
while  the  other  bases  investigated  have  not  reached  an  acceptable  level  of  use  for  any  measurable 
length  of  time.  Because  Randolph  had  the  most  experience  in  actually  using  the  system,  the  base 
was  assumed  to  have  the  most  reliable  data  attributable  to  POMX  data  entry.  The  categorization 
of  the  data  is  further  explored  in  the  next  section. 

Data  Organization 

To  make  comparisons  among  the  data  provided  by  POMX  or  traditional  CAMS  data 
entry,  it  is  necessary  to  code  the  data  in  a  meaningful  format.  To  effectively  accomplish  this 
task,  it  is  important  to  determine  which  data  elements  comprise  the  dependent  and  independent 
variables  of  interest  in  this  study.  The  variables  used  to  answer  IQ1  are  categorized  in  Table  4. 


Table  4.  IQ 


Independent  Variable 

Dependent  Variable 

POMX  Status 

Error  Rate 

Variables 


Error  rate  is  computed  as  shown  in  equation  (1). 

#  Jobs  in  Error 

hK  — 


#  Jobs  Reviewed 


(1) 


where 

ER  =  Error  Rate 

Answering  IQ2  involves  analyzing  different  types  of  discrepancies  that  lead  to  an 
individual  job  being  counted  as  an  error.  This  study  evaluates  the  following  five  discrepancies: 


Table  5.  IQ2  Discrepancy  Types 


Abbreviation 

Discrepancy 

wuc 

Work  Unit  Code 

AT 

Action  Taken  Code 

HMAL 

How  Malfunctioned  Code 

WD 

When  Discovered  Code 

WCE  NARR 

Work  Center  Event  Narrative 
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Whereas  in  IQ  1  a  single  error  rate  is  evaluated,  in  IQ2  discrepancy  rate  by  POMX  status  is 
evaluated  for  each  of  the  five  discrepancy  types.  The  variables  of  interest  are  shown  in  Table  6. 


Table  6.  IQ2  Variables 


Independent  Variables 

Dependent  Variable 

POMX  Status 

#  of  Discrepancies 

This  categorization  allows  for  comparison  of  the  dependent  variable  “number  of  discrepancies” 
based  on  the  matrix  shown  in  Table  7  to  determine  if  there  are  significant  differences  between 
the  types  of  discrepancies  identified  in  Table  5. 


Table  7.  IQ2  Comparison  Matrix 


Construct 

Sample  1 

Sample  2 

1 

NON  POMX,  AT 

POMX,  AT 

2 

NON  POMX,  WUC 

POMX,  WUC 

3 

NON  POMX,  HMAL 

POMX,  HMAL 

4 

NON  POMX,  WD 

POMX,  WD 

5 

NON  POMX,  WCE  NARR 

POMX,  WCE  NARR 

The  data  of  interest  include  the  monthly  error  rates  and  number  of  errors  before  and  after 
POMX  implementation.  A  period  of  24  months  before  POMX  implementation  comprises 
Sample  1,  and  24  months  after  implementation  comprises  Sample  2.  The  use  of  equal  sample 
sizes  ensures  a  balanced  experimental  design  and  allows  more  flexibility  in  the  data  analysis. 
Data  from  June  and  July  of  2003  are  excluded  as  this  was  the  transition  period  during  which 
Randolph  was  installing  the  POMX  infrastructure  and  implementing  the  system.  Due  to  the 
small  sample  sizes,  the  t-statistic  is  appropriate  to  test  for  significant  difference  between  the 
mean  error  rates  of  the  two  samples.  This  is  necessary  because  we  are  unable  to  rely  on  the 
Central  Limit  Theorem  which  would  allow  us  to  assume  the  data  were  normally  distributed  if 
both  sample  sizes  contained  at  least  30  measurements.  The  comparisons  in  IQ2  are  made  only 
between  sample  groups  that  include  the  same  type  of  discrepancy.  The  logic  for  this  decision 
lies  in  the  fact  that  this  research  is  not  concerned  with  the  frequency  of  types  of  discrepancies  in 
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comparison  with  the  frequency  of  other  types  of  discrepancies;  rather  this  study  remains  focused 
solely  on  determining  whether  the  POMX  variable  has  any  effect  on  these  five  different  types  of 
discrepancies. 

Hypotheses  and  Analysis 

This  section  details  the  specific  hypotheses  used  to  test  the  investigative  questions. 
Equations  (2)  and  (3)  are  provided  to  demonstrate  the  theory  behind  the  analytical  test;  however 
the  actual  tests  are  conducted  using  the  SAS  JMP  6.0  statistical  software  package.  Following 
traditional  convention,  H0  refers  to  the  null  hypotheses  and  Ha  refers  to  the  alternative 
hypothesis.  The  following  hypothesis  is  proposed  to  test  IQ1 : 


Hypothesis  1:  H0:  p?  >  p i 
Ha:  P2<  Pi 

where  pi  =  mean  population  error  rate  without  POMX 
and  p2  =  mean  population  error  rate  with  POMX 

This  hypothesis  is  tested  using  the  t-statistic  at  the  .05  level  of  significance.  A  one-way  test  is 

used  because  this  research  is  interested  in  whether  the  POMX  mean  value  is  lower  than  the  NON 


POMX  mean  value.  The  formula  for  computing  the  critical  value  of  t  is  shown  in  equation  (2). 


t  = 


(X,-X2) 


2.  1  1 


) 


(2) 


where 

Xl  =  sample  1  mean 
and  X2  =  sample  2  mean 
and  s2p  =  pooled  sample  estimator  of  a2 
and  =  number  of  measurements  in  sample  1 

and  ft2  =  number  of  measurements  in  sample  2 
(adapted  from  McClave  et  ah,  2005) 
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The  value  s2  is  a  pooled  sample  estimator  of  cr  (the  population  variance).  Since  the  use  of  the 
t-statistic  assumes  equal  variance  between  the  two  populations,  the  sample  variances  can  be 
pooled  to  provide  a  better  estimate  of  a2  as  shown  in  equation  (3)  (McClave  et  ah,  2005). 


nx  +  n2-  2 


(3) 


where 

Xx  =  a  measurement  from  sample  1 
and  X2  =  a  measurement  from  sample  2 
and  Xx  =  mean  from  sample  1 
and  X2  =  mean  from  sample  2 
and  nx  =  number  of  measurements  in  sample  1 
and  n2  =  number  of  measurements  in  sample  2 
(adapted  from  McClave  et  ah,  2005) 

There  are  some  assumptions  that  are  generally  accepted  as  necessary  for  the  t-statistic  to  yield 
valid  results.  Both  populations  must  be  approximately  normally  distributed,  and  the  population 
variances  should  be  equal  (McClave  et  ah,  2005).  Normality  is  tested  using  the  “fit  distribution” 
and  “goodness  of  fit  test”  functions  in  JMP  6.0  at  a  .05  level  of  significance.  Equality  of 
variances  is  tested  using  the  “Levene  test”  function  in  JMP  6.0  at  the  .05  level  of  significance. 

In  some  instances  the  populations  do  not  meet  the  required  assumptions  and  adjustments 
must  be  made  to  the  test  procedures.  While  in  some  cases  the  problem  can  be  simply  noted  and 
the  standard  t-test  used  anyway,  other  situations  require  further  explanation  and  adjustments  to 
the  formula  used  to  compute  the  critical  value  of  the  test  statistic.  In  some  instances  increased 
reliance  in  placed  on  other  research  in  statistical  methods  to  help  deal  with  the  violation  of 


assumptions.  Any  such  departures  from  the  nonn  are  noted  in  the  appropriate  section  of  the 


results  and  analysis  of  the  next  chapter. 
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The  same  procedures  used  for  testing  IQ1  are  used  to  test  hypotheses  2  -  6  for  IQ2. 

Hypothesis  2\  H0:  p2>pi 
Ha:  p2<  Pi 

where  pi  =  mean  #  of  Action  Taken  (AT)  code  discrepancies  without  POMX 
and  P2  =  mean  #  of  Action  Taken  (AT)  code  discrepancies  with  POMX 

Hypothesis  3:  H0:  p2>pi 
Ha:  p2<  pi 

where  pi  =  mean  #  of  Work  Unit  Code  (WUC)  discrepancies  without  POMX 
and  p2  =  mean  #  of  Work  Unit  Code  (WUC)  discrepancies  with  POMX 

Hypothesis  4\  H0:  p2>pi 
Ha:  p2<  pi 

where  pi  =  mean  #  of  How  Malfunction  (HMAL)  code  discrepancies  without  POMX 
and  p2  =  mean  #  of  How  Malfunction  (HMAL)  code  discrepancies  with  POMX 

Hypothesis  5:  H0:  p2>pi 
Ha:  p2<  pi 

where  pi  =  mean  #  of  When  Discovered  (WD)  discrepancies  without  POMX 
and  p2  =  mean  #  of  When  Discovered  (WD)  discrepancies  with  POMX 

Hypothesis  6:  H0:  p2>pi 
Ha:  p2<  pi 

where  pi  =  mean  #  of  Work  Center  Event  Narrative  (WCE  NARR)  discrepancies  without  POMX 
and  p2  =  mean  #  of  Work  Center  Event  Narrative  (WCE  NARR)  discrepancies  with  POMX 

Summary 

This  chapter  detailed  the  analysis  used  in  the  analysis  of  POMX  error  rates  at  Randolph 
AFB,  TX.  After  reviewing  the  research  problem  and  investigative  question,  the  major 
assumptions  were  stated  regarding  the  use  of  the  POMX  system  at  Randolph  and  the  exclusion 
of  other  bases  which  did  not  have  sufficient  data  for  this  study.  Next  was  a  discussion  of  the 
organization  and  coding  of  the  data,  which  included  the  identification  of  the  independent  and 
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dependent  variables.  The  chapter  concluded  by  addressing  the  specific  hypotheses  and  test 
statistic  used  to  make  the  comparisons  between  the  samples.  The  next  chapter  applies  these 
methods  and  captures  the  results  from  the  hypothesis  testing. 
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IV.  Results  and  Analysis 


Chapter  Overview 

This  chapter  reports  the  results  of  the  data  analysis  conducted  as  part  of  the  research 
design  described  in  the  previous  chapter.  Each  test  of  hypothesis  is  treated  individually, 
followed  by  a  synthesis  of  the  results  as  they  pertain  to  the  investigative  questions.  The 
populations  from  which  the  POMX  and  NON  POMX  samples  are  taken  are  assumed  to  be 
approximately  normally  distributed  unless  otherwise  indicated  by  the  results  of  the  Shapiro-Wilk 
test.  A  test  of  equal  variances  for  the  samples  being  compared  in  each  hypothesis  is  also 
conducted  to  ensure  the  proper  use  of  the  test  statistic  and  the  validity  of  the  results.  The  data  in 
this  study  were  analyzed  using  the  SAS  JMP  6.0  statistical  software  package.  The  results  for  the 
tests  of  the  normality  and  variance  assumptions  are  available  in  Appendix  C. 

Test  of  Hypothesis  1:  H0:  p2>pi 

Ha:  p2<  pi 

where  pi  =  mean  population  error  rate  without  POMX 
and  p2  =  mean  population  error  rate  with  POMX 

The  assumption  of  a  normally  distributed  population  in  the  NON  POMX  data  set  was  not 
entirely  satisfied  by  the  Shaprio-Wilk  goodness-of-fit  test  in  the  sample  data.  The  months  of 
December  2000  and  March  2002  appear  to  be  outliers  in  relation  to  the  remainder  of  the 
distribution.  When  these  outliers  were  removed,  the  sample  distribution  was  still  not  quite 
nonnal.  For  the  purposes  of  the  test  the  outliers  remained  in  the  distribution  to  ensure  the 
maximum  amount  of  available  data  was  considered  in  the  means  comparison,  and  the  true 
population  was  assumed  to  be  normally  distributed.  The  Levene  test  for  equal  variances  between 
the  POMX  and  NON  POMX  samples  indicated  that  they  are  equal. 
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Based  on  the  results  in  Table  8,  there  was  no  evidence  that  the  POMX  error  rate  was  less 


than  the  NON  POMX  error  rate. 

_ Table  8,  Hypothesis  1  Results 

Levene  test  for  Equal  Variances _ 

p-value  =  0.9115 

Variances  are  equal 


POMX  Status 

N 

Mean  (p) 

Std.  Dev.  (a) 

NON  POMX 

24 

0.2163 

0.1577 

POMX 

24 

0.2518 

0.1348 

t-test _ 

p-value  =  0.2031 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 

Test  of  Hypothesis  2:  H0:  p2>pi 

Ha:  p2<  pi 

where  pi  =  mean  #  of  Action  Taken  (AT)  code  discrepancies  without  POMX 
and  p2  =  mean  #  of  Action  Taken  (AT)  code  discrepancies  with  POMX 

In  this  case,  the  normality  assumptions  were  not  met  for  either  sample;  however  the 

distributions  did  possess  equal  variances  and  were  symmetric.  According  to  his  review  of  Clinch 

&  Kessehnan  (1982)  and  Tan  (1982),  George  Keppel  noted  that  the  results  of  the  F-test  (or  t-test 

in  this  case)  are  not  especially  affected  by  departures  from  normality  as  long  as  the  distributions 

are  symmetric  and  of  equal  sample  size  greater  than  n  =  12  (Keppel,  1991).  Because  the  number 

of  measurements  in  each  sample  was  equal,  the  distributions  possessed  equal  variances,  and  the 

distributions  were  symmetric,  the  departure  from  normality  was  not  considered  to  be  a 

significant  barrier  to  the  validity  of  the  test.  This  foundation  was  used  in  subsequent  hypothesis 

testing  when  similar  conditions  existed.  Based  on  the  results  in  Table  9,  there  was  no  evidence 
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that  the  mean  number  of  POMX  AT  discrepancies  was  less  than  the  mean  number  of  NON 
POMX  AT  discrepancies. 

_ Table  9.  Hypothesis  2  Results _ 

Levene  test  for  Equal  Variances _ 

p-value  =  0.7909 

Variances  are  equal 


POMX  Status 

N 

Mean  (u)  Std.  Dev.  (s) 

NON  POMX 

24 

0.875 

1.227 

POMX 

24 

1.292 

1.1971 

t-test 

p-value  = 

0.1199 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 

Test  of  Hypothesis  3:  H0:  p2>pi 

Ha:  p2<  pi 

where  pi  =  mean  #  of  Work  Unit  Code  (WUC)  discrepancies  without  POMX 
and  p2  =  mean  #  of  Work  Unit  Code  (WUC)  discrepancies  with  POMX 

Again,  the  Shipro-Wilk  test  showed  that  the  NON  POMX  sample  data  were  not  normally 

distributed;  however  the  graph  in  Appendix  C  shows  that  the  data  resembled  a  normal 

distribution  more  than  other  types  of  distributions.  It  is  believed  that  a  larger  sample  size  would 

be  normally  distributed,  therefore  the  population  data  were  assumed  to  be  normally  distributed 

and  no  further  modification  was  made  for  the  purposes  of  comparing  the  mean  WUC 

discrepancy  rates.  The  Levene  test  indicated  that  the  variances  were  also  unequal  in  these 

samples;  therefore  the  t-statistic  was  adjusted  by  modifying  the  degrees  of  freedom  used  to 

compute  the  critical  t-value  (McClave  et  ah,  2005). 
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The  resulting  formulas  are  shown  in  equations  (4)  and  (5). 


t  = 


A+A 


n. 


nn 


(4) 


where 

Xx  =  sample  1  mean 
and  X2  =  sample  2  mean 
and  .s’,2  =  sample  estimator  of  of 
and  si  =  sample  estimator  of  of 
and  nx  =  number  of  measurements  in  sample  1 
and  n2  =  number  of  measurements  in  sample  2 
(adapted  from  McClave  et  ah,  2005) 

The  difference  between  equations  (2)  (used  to  test  hypotheses  1  &  2)  and  (4)  is  that  equation  (2) 
pools  the  variances  from  both  samples  to  compute  the  critical  value  of  the  t-statistic  while 
equation  (4)  keeps  the  sample  variances  separate.  Additionally,  equation  (5)  must  be  used  to 
adjust  the  degrees  of  freedom  to  account  for  the  difference  in  sample  sizes.  The  computed  value 


is  rounded  down  to  the  nearest  integer  for  use  in  selecting  the  proper  critical  value  from  the 
t-table  (McClave  et  al.,  2005). 


2  2 

(^  +  ^)2 

>h  fh 

2  2 

(— )2  (^)2 
»i  !  n2 
nx  - 1  n2-l 


(5) 


where 


v  =  the  degrees  of  freedom  used  to  compute  the  critical  value  of  t 
and  .s',2  =  sample  estimator  of  of 

and  si  =  sample  estimator  of  of 

and  nx  =  number  of  measurements  in  sample  1 

and  n2  =  number  of  measurements  in  sample  2 

(adapted  from  McClave  et  ah,  2005) 
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Based  on  the  results  in  Table  10,  there  was  no  evidence  that  the  mean  number  of  POMX  WUC 


discrepancies  was  less  than  the  mean  number  of  NON  POMX  WUC  discrepancies. 

_ Table  10.  Hypothesis  3  Results _ 


Levene  test  for  Equal  Variances 

p-value  = 

0.3884 

Variances  are  not  equal 

POMX  Status 

N 

Mean  (p) 

Std.  Dev.  (a) 

NON  POMX 

24 

2.125 

1.8253 

POMX 

24 

1.9583 

1.4289 

t-test 

p-value  = 

0.3631 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 

Test  of  Hypothesis  4:  H0:  p2>Pi 

Ha:  p2<  pi 

where  pi  =  mean  #  of  How  Malfunction  (HMAL)  code  discrepancies  without  POMX 
and  p2  =  mean  #  of  How  Malfunction  (HMAL)  code  discrepancies  with  POMX 

The  Shapiro-Wilk  test  indicated  that  neither  the  NON  POMX  nor  the  POMX  sample  data 

were  normally  distributed.  This  test  of  hypothesis  assumed  that  the  population  is  normally 

distributed  based  on  the  shape  of  the  distribution  even  though  the  results  of  the  goodness-of-fit 

test  indicated  otherwise.  It  is  believed  that  a  larger  sample  size  would  support  this  conclusion. 

The  Levene  test  indicated  that  the  variances  between  the  two  sample  distributions  were  not 

equal;  therefore  the  t-statistic  was  adjusted  by  using  equations  (4)  and  (5)  to  modify  the  degrees 

of  freedom  as  demonstrated  in  the  previous  discussion  of  the  hypothesis  3  test  results.  Based  on 

the  results  in  Table  11,  there  was  no  evidence  that  the  mean  number  of  POMX  HMAL 

discrepancies  was  less  than  the  mean  number  of  NON  POMX  HMAL  discrepancies 
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Table  11.  Hypothesis  4  Results 


Levene  test  for  Equal  Variances 

p-value  = 

Variances  are  not  equal 

0.0006 

POMX  Status 

N 

Mean  (p) 

Std.  Dev.  (a) 

NON  POMX 

24 

0.9583 

0.9079 

POMX 

24 

3.6667 

2.8387 

t-test _ 

p-value  =  0.9999 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 

Test  of  Hypothesis  5:  H0:  p2>Pi 

Ha:  \i2<  pi 

where  pi  =  mean  #  of  When  Discovered  (WD)  discrepancies  without  POMX 
and  p2  =  mean  #  of  When  Discovered  (WD)  discrepancies  with  POMX 

The  Shapiro-Wilk  test  indicated  that  the  data  in  both  samples  were  not  normally 

distributed.  The  distributions  were  symmetric,  and  the  Levene  test  indicated  equal  variances. 

Using  the  logic  from  hypothesis  2,  the  departure  from  normality  was  not  thought  to  impact  the 

test  results.  Based  on  the  results  in  Table  12,  there  was  no  evidence  that  the  mean  number  of 

POMX  WD  discrepancies  was  less  than  the  mean  number  of  NON  POMX  WD  discrepancies. 

_ Table  12.  Hypothesis  5  Results _ 

Levene  test  for  Equal  Variances _ 

p-value  =  0.9119 

Variances  are  equal 


POMX  Status 

N 

Mean  (p) 

Std.  Dev.  (a) 

NON  POMX 

24 

0.125 

0.3378 

POMX 

24 

0.125 

0.4484 

t-test _ 

p-value  =  0.5 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 


35 


Test  of  Hypothesis  6:  H0:  p2>Pi 

Ha:  p2<  pi 

where  pi  =  mean  #  of  Work  Center  Event  Narrative  (WCE  NARR)  discrepancies  without  POMX 
and  p2  =  mean  #  of  Work  Center  Event  Narrative  (WCE  NARR)  discrepancies  with  POMX 

The  data  in  both  samples  were  not  normally  distributed,  and  the  variances  were  not  equal. 

In  previous  tests  of  hypothesis  this  research  made  assumptions  regarding  the  nonnal  distribution 

of  the  populations  from  which  the  data  was  gathered,  however  it  did  not  seem  appropriate  in  this 

case.  The  degree  to  which  the  data  were  not  normally  distributed  seemed  to  be  significant  based 

on  the  graphs  of  the  data  and  the  results  of  the  Shapiro-Wilk  tests.  The  Wilcoxon  Rank  Sum  test 

is  a  nonparametric  way  of  determining  whether  the  populations  possess  equivalent  distributions 

when  the  assumptions  of  normality  and  equal  variances  are  not  met.  Based  on  the  results  in 

Table  13,  there  was  no  evidence  that  the  number  of  POMX  WCE  NARR  discrepancies  was  less 

than  the  number  of  NON  POMX  WCE  NARR  discrepancies. 


Table  13.  Hypothesis  6  Results 


Levene  test  for  Equal  Variances 

p-value  = 

0.0157 

Variances  are  not  equal 

POMX  Status 

N 

Mean  (p) 

Std.  Dev.  (a) 

NON  POMX 

24 

1.2083 

3.323 

POMX 

24 

0.2917 

0.55 

Wilcoxon  Ranked  Sums  Test 

p-value  = 

0.3929 

Fail  to  reject  Ho 

POMX  is  greater  than  or  equal  to  NON  POMX 
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Summary  of  Quantitative  Results 


Table  14  shows  that  no  test  of  hypothesis  indicated  improvement  in  data  accuracy  using 


POMX  compared  to  using  data  entry  methods  away  from  the  point-of-maintenance  at  Randolph 


AFB  during  the  48  months  that  comprised  the  data  samples. 


Table  14.  Summary  of  Hypothesis  Test  Results 

Hypothesis  Discrepancy  Type  Comparison  Results 


1 

Overall  Error  Rate 

POMX 

> 

NON  POMX 

2 

AT 

POMX 

> 

NON  POMX 

3 

WUC 

POMX 

> 

NON  POMX 

4 

HMAL 

POMX 

> 

NON  POMX 

5 

WD 

POMX 

> 

NON  POMX 

6 

WCE NARR 

POMX 

> 

NON  POMX 

These  results  did  not  batch  expectations.  It  was  suspected  that  POMX  would  lead  to  decreased 
error  rates  over  conventional  methods  of  inputting  data  because  the  data  would  be  captured 
immediately  following  the  completion  of  the  maintenance  action.  Because  no  difference  was 
found  between  the  two  groups,  the  following  post  hoc  user  interviews  attempted  to  capture  the 
reasons  why  POMX  did  not  lead  to  increased  data  accuracy  at  Randolph  AFB. 

Post  hoc  Interviews 

Further  exploration  into  possible  causes  for  the  lack  of  increased  data  accuracy  revealed 
that  POMX  was  not  being  used  at  Randolph  AFB  to  the  extent  originally  thought  at  the  onset  of 
this  research.  A  similar  finding  occurred  earlier  during  this  study  regarding  POMX  use  at 
Langley  and  Nellis.  Though  higher  organizational  echelons  (above  Wing  level)  believed  there 
was  a  moderate  amount  of  system  use,  phone  calls  to  the  using  units  revealed  otherwise.  This 
disparity  drove  the  need  to  conduct  a  series  of  interviews  at  the  unit  level  in  an  effort  to  capture  a 
broad  understanding  of  what  was  actually  occurring  at  these  bases,  and  what  factors  were  behind 
the  minimal  system  use. 
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Randolph,  Nellis,  and  Langley  were  selected  as  the  three  focal  bases  for  these  interviews, 
and  an  additional  interview  was  conducted  with  an  individual  at  Hurlburt  Field,  Florida 
regarding  the  POMX  version  3.0  effort  sponsored  by  AF/A4MM.  These  bases  were  selected 
because  they  were  the  bases  originally  thought  to  have  robust  POMX  systems  that  had  been 
successfully  implemented  and  were  receiving  high  levels  of  usage.  It  was  from  these  bases  that 
data  was  originally  sought  for  the  quantitative  analysis  portion  of  this  research  before  it  was 
discovered  that  the  data  was  not  representative  of  actual  POMX  use. 

Fifteen  individuals  were  contacted  and  asked  to  answer  three  open-ended  questions 
regarding  their  experience  with  POMX  implementation  at  their  respective  bases.  Twelve 
respondents  provided  their  observations  via  email  or  telephone,  and  this  equated  to  an  80% 
response  rate.  The  respondents  are  coded  to  ensure  anonymity,  and  each  base  is  treated  as  a 
separate  case.  The  respondents  are  coded  as  shown  in  Table  15. 

Table  15.  List  of  Respondents 
Code  Respondent 

LI  Respondent  1  at  Langley  AFB 

L2  Respondent  2  at  Langley  AFB 

N1  Respondent  1  at  Nellis  AFB 

N2  Respondent  2  at  Nellis  AFB 

N3  Respondent  3  at  Nellis  AFB 

N4  Respondent  4  at  Nellis  AFB 

N5  Respondent  5  at  Nellis  AFB 

R1  Respondent  1  at  Randolph  AFB 

R2  Respondent  2  at  Randolph  AFB 

R3  Respondent  3  at  Randolph  AFB 

R4  Respondent  4  at  Randolph  AFB 

HI  Respondent  1  at  Hurlburt  AFB 


Two  respondents  were  from  Langley  and  Nellis  F-22  units  that  use  the  Integrated  Maintenance 
Information  System  (IMIS),  a  proprietary  system  designed  by  Lockheed  Martin  to  perform 
maintenance  data  collection  functions  for  the  F-22  Raptor.  While  this  system  does  not  utilize 
POMX  or  IMDS  directly,  the  concept  of  capturing  data  at  the  point  of  maintenance  is  the  same 
as  the  other  bases,  and  many  of  the  findings  are  just  as  relevant.  The  F-22  responses  are 
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incorporated  among  the  POMX  responses,  and  the  IMIS  specific  infonnation  is  included  due  to 
the  fact  that  some  readers  may  find  the  insights  meaningful.  Finally,  some  responses  were  edited 
for  clarity  while  remaining  diligent  to  capture  the  original  thought. 

Langley  AFB  Responses 

The  respondents  from  Langley  AFB  included  representatives  from  the  maintenance 
analysis  and  F-22  units. 

Question  1:  What  efficiencies  have  been  experienced  by  using  the  POMX  or  IMIS  systems  when 
they  operate  as  they  should?  (How  do  they  make  life  better  than  the  old  way  of  taking  time  in 
CAMS/IMDS?  For  IMIS,  discuss  batch  mode  if  unable  to  use  the  RF  LAN). 

LI:  Good  when  it  works;  saves  time  to  and  from  the  building. 

L2:  The  ability  to  update  and  retrieve  maintenance  data,  and  to  research 

aircraft  history  from  the  flightline,  is  beneficial.  The  Windows  XP  environment  is 
user  friendly  and  familiar  to  most  users.  Personnel  can  access  the  Air  Force 
portal  during  breaks  or  lulls  to  review  personnel  data  or  review  Air  Force 
Instructions  (AFIs). 

Question  2:  What  were  the  implementation  barriers?  (What  slowed  you  down  or  kept  you  from 
using  the  system?  What  problems  existed/still  exist?) 

LI:  Technical  Barriers:  The  data  transfer  process  from  IMIS  to  IMDS  is  slow 

and  sometimes  locks  the  system  up  for  30  -  40  minutes.  When  this  happens, 
personnel  are  unable  to  take  any  further  action  until  the  system  completes  the 
transfer. 

Workforce  Barriers:  Workers  become  frustrated  with  the  slow  system  and 
when  the  system  times  out  and  logs  them  off  involuntarily.  Rather  than  waste 
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their  time  only  to  get  logged  off  before  they  complete  their  documentation,  many 
personnel  prefer  to  go  into  the  building  and  use  the  wired  terminal. 

Management/Leadership  Barriers:  POMX  came  suddenly  and  there  did 
not  seem  to  be  an  appropriate  amount  of  preparation.  It  was  uncertain  who  in  the 
Maintenance  Group  would  be  in  charge  of  the  effort.  Effective  coordination  of 
schedules  and  timelines  was  an  issue,  and  there  was  little  direction  in  how  to 
prepare  for  the  implementation.  The  Communications  Squadron  (Comm.)  is 
charged  with  the  administration  of  the  wireless  network  that  enables  POMX; 
however  clear  communication  is  lacking  between  Comm,  and  the  users.  Comm, 
personnel  do  not  seem  to  fully  understand  the  needs  of  the  maintenance  personnel 
and  how  the  system  is  used. 

Technical  Barriers:  Network  connectivity  continues  to  be  a  problem.  If 
too  many  users  are  on  the  wireless  network  at  one  time,  then  some  users  are 
dropped  without  warning. 

Workforce  Barriers:  The  contractor  provided  limited  training  in  the  use 
and  maintenance  of  the  system.  Information  Management  (IM)  personnel  at  the 
flight  level  are  overtasked.  POMX  did  not  replace  any  of  the  IMs  former  duties; 
rather  it  added  tasks  to  an  already  busy  career  field  without  additional  manpower 
provisions.  A  single  contractor  representative  is  on  base,  and  he  resides  within 
the  Comm,  squadron,  not  within  the  aircraft  maintenance  squadron.  When  the 
IMs  have  a  problem  with  the  system,  their  lack  of  training  requires  them  to  track 
down  the  contractor  representative. 


What  needs  to  be  different  in  future  POMX  implementations? 

Technical:  The  network  coverage  needs  to  be  comprehensive  from  the 
beginning.  The  IMIS  software  user  interface  could  also  be  improved. 

Management/Leadership:  Lines  of  communication  between  the 
maintenance  organization  and  the  Comm,  squadron  should  be  open  and 
established  early  on,  and  Comm,  needs  to  fully  understand  the  needs  of  the 
maintenance  users. 

Workforce:  Additional  IM  personnel  are  needed,  and  they  need  more 
rigorous  training  in  the  administration  of  the  POMX  system. 

Nellis  AFB  Responses 

The  respondents  from  Nellis  AFB  included  representatives  from  F-22,  F-15,  F-16,  and 
A- 10  aircraft  maintenance  units  (AMUs). 

Question  1:  What  efficiencies  have  been  experienced  by  using  the  POMX  or  IMIS  systems  when 
they  operate  as  they  should?  (How  do  they  make  life  better  than  the  old  way  of  taking  time  in 
CAMS/IMDS?  For  IMIS,  discuss  batch  mode  if  unable  to  use  the  RF  LAN). 

Nl:  The  ability  to  immediately  capture  data  as  it  happens  and  complete  the 

process  at  the  job  site  improves  the  accuracy  of  the  data  (as  opposed  to  the  old 
way  of  having  one  individual  accomplish  all  the  documentation  for  the  entire 
team  regardless  of  whether  they  perfonned  the  actual  maintenance  themselves) 
and  allows  for  less  end  of  shift  turnover  and  wrap-up  time.  Additionally,  the 
ability  to  access  maintenance  data  at  the  aircraft  saves  travel  time  of  going  back  to 
the  shop. 


Question  3: 

LI: 


L2: 
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N2:  POMX  provides  near  real  time  CAMS/IMDS  update  capability.  This 

allows  the  maintainer  to  open  and  close  jobs  at  the  work  site  rather  than  having  to 
find  a  hardwired  desktop  CAMS  terminal  to  perform  this  task.  Technicians  can 
order  parts  through  SBSS  without  leaving  job  site.  Expeditors  and  production 
superintendents  can  verify  CAMS/IMDS  (380  Screen)  against  the  aircraft  781 
series  forms  to  ensure  the  aircraft  is  safe  to  fly.  Maintainers  no  longer  have  to  go 
into  the  office  to  do  CAMS/IMDS  updates  before  they  can  go  home. .  .this  allows 
less  rushing  to  complete  the  documentation  so  they  can  leave  around  the  12  hour 
mark.  The  bottom  line  is  that  with  proper  usage  the  maintainer  should  spend 
more  time  on  his/her  jet  because  the  need  to  leave  the  flightline  to  access 
CAMS/IMDS  has  been  eliminated. 

N3:  The  use  of  the  wireless  gives  us  the  ability  to  complete  maintenance 

actions  on  the  spot  concurrent  with  the  maintenance;  this  alleviates  having  to  send 
someone  inside  once  the  work  is  done  to  complete  the  computer  work.  In 
addition,  some  of  the  actions  we  complete  require  that  we  input  the  part  number 
and  serial  number  of  the  component  at  the  time  of  removal/installation. 

Frequently  we  make  several  trips  back  out  to  the  aircraft  to  confirm  these 
numbers,  but  with  POMX  we  have  the  part  right  in  front  of  us  as  we  enter  the 
information.  From  a  supervisory  standpoint,  it  gives  us  direct  oversight  of 
everything  in  CAMS/IMDS,  as  well  as  in  other  base  tracking  systems,  without 
having  to  leave  the  area.  This  is  very  important  during  the  flying  day  when  we 
must  oversee  CAMS/IMDS  actions,  but  cannot  leave  the  flightline.  The  potential 
is  tremendous  and  we  are  only  beginning  to  see  the  benefits  of  using  the  POMX. 
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N4: 


The  system  works  well  when  you  receive  a  discrepancy  from  pilot  data 
transfer  cartridge  that  has  a  failure  code  attached  to  it.  Jobs  are  automatically 


scheduled  with  job  numbers,  and  the  maintainer  is  able  to  simply  open  the 
electronic  forms,  perform  task,  and  is  taken  to  the  appropriate  electronic  technical 
data  session.  Once  the  job  is  complete,  forms  are  automatically  annotated.  When 
everything  works  right,  the  system  is  very  efficient.  IMIS  makes  the  job  easier 
with  respect  to  time  expended  because  you  are  only  documenting  one  electronic 
maintenance  document  as  opposed  to  paper  forms  and  CAMS/IMDS.  Also,  there 
is  increased  traceability  and  sense  of  responsibility  with  IMIS  because  each 
maintainer  inputs  their  own  data.  Under  the  old  system,  a  single  maintainer 
sometimes  input  the  data  for  all  jobs  on  a  shift  while  other  maintainers  performed 
the  work.  While  this  is  often  someone  who  is  injured,  pregnant,  or  otherwise 
unable  to  work  directly  on  aircraft,  job  data  is  often  diluted,  misunderstood,  or 
otherwise  lost  during  translation. 

N5:  Time  management  efficiencies  are  realized  as  maintainers  no  longer  need 

to  leave  the  flightline  and  travel  to  and  from  a  building  to  perform  the  necessary 
job  data  documentation. 

Question  2:  What  were  the  implementation  barriers?  (What  slowed  you  down  or  kept  you  from 
using  the  system?  What  problems  existed/still  exist?) 

Nl:  Technical  Barriers:  Network  coverage  is  a  problem.  There  are  places  on 

the  flightline  where  the  maintainer  is  working  and  cannot  connect  to  the  network; 
however  a  recent  upgrade  is  expected  to  have  improved  this. 

Workforce  Barriers:  Since  POMX  usage  is  not  absolutely  required,  there 
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are  still  maintainers  who  do  not  use  it  and  are  not  even  trained  on  how  to  use  it. 


Maintainers  are  very  hesitant  to  use  the  WLAN  when  faced  with  the  loss  of  data 
and  the  need  to  reaccomplish  their  documentation  work. 

N2:  Management/Leadership  Barriers:  Buy-in  from  the  rank  and  file  wasn’t 

solicited.  It  came  down  with  a  “this  is  how  you  use  it  and  you  WILL  use  it” 
methodology.  Management  help  with  overcoming  funding  procurement  issues 
with  storage  racks  for  the  hardware  was  problematic.  The  fielding  plan  was  not 
well  developed  as  care  and  feeding  of  the  system  was  an  afterthought. 
Requirements  were  not  effectively  pushed  down  to  personnel.  Laptop  usage  was 
not  strongly  advocated. 

Technical  Barriers:  Output  power/connectivity  was  an  issue  until 
approximately  1  February  2006.  Laptops  would  lose  connection  if  barriers  were 
in  the  way  or  if  signal  strength  became  weak  because  of  distance  to  antennas. 

Low  throughput  was  causing  sluggish  response  when  data  updated  or  transferred. 
The  requirement  to  ensure  laptop  was  logged  into  the  network  at  least  once  a 
week  for  required  security  updates  became  another  task  for  an  already  burdened 
support  section. 

Workforce  Barriers:  Culture  barriers  are  difficult  to  overcome.  Some 
maintainers  are  resistant  to  change. 

N3:  Management/Leadership  Barriers:  Initially  it  was  challenging  getting 

everyone  under  the  mindset  that  these  are  really  computers  that  will  work  on  the 
flightline  just  as  they  do  in  the  building.  Since  that  time  folks  have  started  to  gain 
some  understanding  and  trust,  but  there  is  still  a  lot  of  ground  to  cover.  The 
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biggest  challenge  to  management  is  trying  to  keep  the  workers  interested  in  the 
light  of  the  extensive  downtime  issues.  We  are  just  coming  back  up  from  having 
non-availability  for  over  3  months  and  effectively  have  to  start  over  when  it 
comes  to  inspiring  personnel  to  use  these  computers  enough  to  regain 
competency. 

Technical  Barriers:  System  downtime  and  connectivity  issues  continue  to 
be  a  problem.  The  wireless  signal  on  our  flightline  area  is  very  unreliable  and  has 
led  to  a  great  amount  of  distrust  regarding  use  of  the  laptops. 

Workforce  Barriers:  We  have  no  formal  training  for  using  the  POMX 
laptops  or  for  any  other  PC,  so  the  training  comes  via  trial  and  error  or  asking 
somebody.  Occasionally  this  leads  to  problems  where  folks  inadvertently  cause 
problems,  or  undo  something  that  requires  a  great  amount  of  work  to  correct. 
Maintaining  the  computers  has  been  the  biggest  problem  and  will  probably 
remain  for  some  time.  Our  single  IM  person  is  burdened  with  too  much  to  begin 
with,  and  adding  the  laptops  did  not  seem  to  help  matters.  They  also  seem  to  be 
lacking  the  necessary  training,  and  many  of  our  problems  go  for  weeks  or  months 
before  they  get  resolved.  We  have  no  real  conduit  to  funnel  our  wireless 
problems  to,  and  our  IM  person  is  not  adequately  trained  or  equipped  to  solve 
them.  We  don’t  have  the  IM  manning  to  maintain  and  solve  the  problems  to 
make  this  system  reliable.  Until  our  people  have  a  reliable  system,  it  will  never 
see  high  usage  rates. 

Technical  Barriers:  IMIS  is  unable  to  log  or  document  the  cannibalization 
(CANN)  of  parts.  The  parts  ordering  process  is  only  about  15%  usable,  therefore 


45 


N5: 


Question  3: 

Nl: 


N2: 


N3: 


it  makes  it  extremely  unusable  for  production  to  process  CANN  parts.  Because 
there  is  no  paper  parts  breakdown,  our  only  source  of  ordering  parts  is  via 
hardware  and  software  trees.  If  we  run  into  problems,  it  forces  us  to  use  our 
Lockheed  reps  to  source  the  parts,  and  this  causes  additional  delays. 

Technical  Barriers:  Connectivity  was  an  issue  until  only  recently.  New 
power  boosters  for  the  access  point  transmitters  have  improved  coverage. 

What  needs  to  be  different  in  future  POMX  implementations? 

Management/Leadership:  Mandating  use  will  ensure  that  problems  are 
readily  identified  and  addressed  while  helping  eliminate  workarounds  and  band- 
aid  processes  that  do  not  offer  permanent  solutions. 

Management/Leadership:  Management  needs  to  be  brought  in  early  in 
order  to  “sell  it”  to  the  prospective  users.  The  users  need  to  be  briefed  in  advance 
and  have  the  opportunity  to  see  a  contractor  demonstration  so  they  can  have  the 
experts  answer  their  questions.  In  addition,  an  extensive  fielding  plan  needs  to  be 
developed  to  address  all  areas  of  logistics  bed  down  at  each  location. 

Technical:  All  issues  such  as  power,  connection,  and  throughput  need  to 
be  resolved  up  front  to  allow  users  to  update  the  databases. 

Workforce:  Need  to  keep  the  workforce  informed.  These  kids  aren’t 
dumb  and  if  they  perceive  the  wireless  CAMS/IMDS  interface  to  be  non-user 
friendly  then  they  will  become  disenchanted  and  frustrated  causing  the 
implementation  to  slow  down  or  stop. 

Management/Leadership:  To  make  POMX  work  effectively  will  require 
patience,  people,  and  training.  This  is  not  a  system  that  can  become  fully 
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functional  overnight.  Once  we  built  up  trust  and  confidence  in  the  system,  we 
experienced  great  usage  rates  among  our  personnel.  This  trust  was  lost  when  the 
wireless  signal  became  problematic.  An  effective  system  of  reporting  difficulties 
with  a  quick  reaction  will  go  a  long  way  to  demonstrate  that  there  is  a 
commitment  to  making  the  system  work.  For  now  it  does  not  appear  that  keeping 
this  system  fully  operational  is  a  priority. 

Technical:  This  is  the  hardest  part,  but  reliable  computers  and  a  reliable 
network  are  essential.  The  quickest  way  to  cause  users  to  lose  interest  and  trust  in 
the  system  is  for  it  to  be  unreliable. 

Workforce:  Early  training  and  information  will  generate  interest  among 
users  and  make  them  more  likely  to  use  the  system. 

N4:  No  response  to  this  question. 

N5:  Technical:  Ensure  the  capability  actually  exists  at  the  unit.  A  thorough 

network  site  survey  will  avoid  a  partial  implementation  that’s  full  of  dead  spots 
and  degraded  signal  areas.  Laptops  should  be  as  modem  as  possible  to  ensure  the 
fastest  transactions,  thus  minimizing  both  timeouts  and  network  congestion. 

Workforce:  “Old-heads”  require  extra  attention  to  bring  them  on  board. 
Randolph  AFB  Responses 

The  respondents  from  Randolph  AFB  included  representatives  from  T-l  and  T-38 
AMUs,  as  well  as  quality  assurance  (QA)  personnel. 

Question  1:  What  efficiencies  have  been  experienced  by  using  the  POMX  or  IMIS  systems  when 
they  operate  as  they  should?  (How  do  they  make  life  better  than  the  old  way  of  taking  time  in 
CAMS/IMDS?  For  IMIS,  discuss  batch  mode  if  unable  to  use  the  RF  LAN). 
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Rl:  No  response. 

R2:  It  is  an  advantage  to  be  able  to  enter  job  data  on  the  flightline  as  opposed 

to  returning  to  the  building  and  having  to  find  a  computer  that  is  not  being  used. 
When  it  works,  we  can  access  Air  Force  Instructions  along  with  other  useful 
websites. 

R3:  Using  the  wireless  system,  we  are  able  to  capture  data  sooner,  and  it  tends 

to  be  more  detailed  as  opposed  to  waiting  until  we  return  to  the  building. 

R4:  It  is  a  useful  system  when  it  works. 

Question  2:  What  were  the  implementation  barriers?  (What  slowed  you  down  or  kept  you  from 
using  the  system?  What  problems  existed/still  exist?) 

Rl:  Technical  Barriers:  Connectivity  problems  have  been  the  biggest 

challenge.  Though  network  coverage  is  good  now,  weather  plays  a  factor  in  the 
quality  of  the  signal.  Excessively  hot  temperatures  and  high  winds  can  cause  a 
user  to  lose  the  network  signal. 

Workforce  Barriers:  Users  have  become  frustrated  with  the  network 
problems.  In  addition,  many  maintainers  perceive  the  laptop  as  an  additional 
burden  when  they’re  already  carrying  large  Technical  Order  (T.O.)  binders  and 
tools  to  the  flightline.  It  is  believed  that  once  Interactive  Electronic  Technical 
Manuals  (IETMs)  replace  the  large  paper  T.O.  libraries,  maintainers  will  be  more 
receptive  to  carrying  the  laptop  as  it  will  reduce  their  burden.  Also,  many  users 
take  the  laptop  indoors  to  use  it  so  they  can  take  a  break  and  escape  the  elements. 
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R2:  Management/Leadership  Barriers:  Laptops  were  brought  in  before  the 

system  was  usable.  We  were  just  showed  how  to  log  in,  but  no  other  user  training 
took  place. 

Technical  Barriers:  The  system  took  too  long  to  connect  and  would  often 
timeout,  which  required  you  to  start  over.  There  were  times  when  you  would  get 
dropped  offline  in  the  middle  of  documenting  your  data,  also  requiring  you  to 
start  over.  Batteries  would  not  stay  charged  early  on,  but  this  was  quickly 
resolved. 

Workforce  Barriers:  Although  the  laptop  is  “ruggedized",  it  is  one  more 
thing  we  have  to  worry  about  dropping  and  damaging.  Also,  operational  tempo 
and  mission  requirements  drive  us  to  use  what  works.  When  the  wireless  system 
keeps  dropping  us  off  or  locking  up,  it  becomes  a  waste  of  time.  As  maintainers 
we  do  not  have  the  time  or  personnel  to  fiddle  with  a  system  that  only  works 
sometimes  when  we  can  go  inside,  get  it  done,  and  move  on  to  the  next  job. 

R3:  Technical  Barriers:  Multiple  logons  are  time  consuming  and  require  that 

you  sit  and  wait  for  the  system  to  finish  logging  you  in.  Many  times  we  get 
dropped  and  then  cannot  log  back  in,  forcing  us  to  go  inside  anyway. 

Workforce  Barriers:  Heat  is  an  issue,  and  going  inside  to  accomplish  the 
documentation  also  gives  us  an  opportunity  for  a  break.  Asking  us  to  accomplish 
this  with  the  wireless  laptops  on  the  flightline  creates  the  appearance  that  we  will 
not  be  allowed  to  take  breaks  from  the  heat  throughout  the  day.  Carrying  T.O.s, 
tools,  and  now  laptops  is  too  much  stuff.  Many  people  are  not  even  familiar  with 
CAMS  and  do  not  know  how  to  use  it.  They  rely  on  the  Red  “X”  authority  to 
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Question  3: 

Rl: 


R2: 


clear  their  CAMS  data  and  don’t  see  why  they  should  have  to  do  it  when  it  has  to 
be  signed  off  anyway  by  a  higher  authority.  They  just  let  the  higher  authority  do 
all  of  the  documentation. 

Management/Leadership  Barriers:  No  initial  training  was  provided 

Technical  Barriers:  Multiple  logons  are  time  consuming  and  frustrating. 

It  has  taken  over  a  year  to  resolve  our  wireless  LAN  connectivity  issues. 

Workforce  Barriers:  There  is  not  a  good  point  of  contact  at  the  unit  level 
for  when  we  have  problems.  Many  people  in  the  unit  weren’t  computer  literate  to 
begin  with  and  were  just  expected  to  use  the  technology. 

What  needs  to  be  different  in  future  POMX  implementations? 

Management/Leadership:  Explain  to  the  workforce  the  importance  of 
using  the  system  before  and  throughout  the  implementation  process. 

Technical:  More  thorough  site  surveys  are  needed  to  ensure  the  proper 
amount  of  access  points  are  designed  into  the  network  architecture  before  turning 
it  over  to  the  user. 

Workforce:  The  most  important  thing  that  can  be  done  to  prepare  the 
workforce  is  to  provide  training  prior  to  expecting  them  to  use  the  system  in  their 
daily  routines. 

Management/Leadership:  There  needs  to  be  more  testing  before  you  put 
this  type  of  system  in  the  user’s  hands. 

Technical:  The  system  needs  good  network  coverage  from  the  beginning 
and  fewer  passwords/logons.  The  ability  to  switch  screens  back  and  forth  like  in 
CAMS  page  fdes  would  be  useful. 
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Workforce:  The  contractor  needs  to  work  more  closely  with  the  users,  and 
perhaps  staff  a  dedicated  on-site  user  representative  in  the  maintenance 
organization  for  a  period  of  time  until  usage  stabilizes. 

R3:  Management/Leadership:  Leadership  needs  to  emphasize  CAMS  usage 

and  documentation  for  all  individuals.  Individuals  who  do  not  even  know  how  to 
operate  CAMS  need  to  be  identified  and  trained. 

Technical:  It  might  be  easier  to  use  the  CAC  card  to  login  if  this  would 
reduce  the  number  of  passwords  we  had  to  remember. 

R4:  Management/Leadership:  Handouts  or  quick  reference  cards  for  using  the 

system  would  be  useful.  Points  of  contact  for  problems  need  to  be  identified  and 
available  to  work  problems  at  the  unit  level. 

Technical:  Digital  T.O.s  would  be  nice,  but  there  is  still  a  need  for  paper 
schematics  and  wiring  diagrams,  as  the  laptop  screen  is  not  large  enough  for 
detailed  wire-tracing. 

Hurlburt  Field  Responses 

The  representative  from  Hurlburt  Field  was  a  program  integrator  familiar  with  the 
ongoing  POMX  effort  and  the  previous  user  evaluations  that  identified  the  software  problems 
currently  being  addressed. 

Question  1:  What  efficiencies  have  been  experienced  by  using  the  POMX  or  IMIS  systems  when 
they  operate  as  they  should?  (How  do  they  make  life  better  than  the  old  way  of  taking  time  in 
CAMS/IMDS?  For  IMIS,  discuss  batch  mode  if  unable  to  use  the  RF  LAN). 

HI:  Air  Force  Special  Operations  Command  (AFSOC)  has  not  seen  the  full 

efficiencies  of  the  POMX  program  due  to  software  problems  during  our  testing. 
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Both  of  the  previous  versions  of  software  had  significant  issues  that  prevented  full 
scale  use  of  the  software.  The  POMX  concept  is  a  sound  one  however.  While 
there  are  no  definitive  numbers,  it  is  anticipated  that  POMX  should  increase  data 
integrity  by  as  much  as  75%  or  more. 

Question  2:  What  were  the  implementation  barriers?  (What  slowed  you  down  or  kept  you  from 
using  the  system?  What  problems  existed/still  exist?) 

HI:  Management/Leadership  Barriers:  Leadership  and  users  are  both 

frustrated  with  POMX  because  of  software  issues.  Leadership  continuously  asks 
“Why  are  we  putting  the  extra  burden  on  the  maintainer  if  the  software  doesn’t 
work  correctly?” 

Technical  Barriers:  Two  main  issues  arose  with  the  use  of  POMX. 

1 .  Wireless  LAN  (WLAN).  Initially  POMX  had  connectivity  issues.  This  turned 
out  to  be  more  configuration  issues  as  opposed  to  the  POMX  software.  This  issue 
did  create  a  major  negative  view  that  POMX  would  not  work  unless  there  was 
connectivity.  This  is  not  the  case  as  the  POMX  software  does  have  a  store 
forward  capability  which  allows  users  to  input  data  and  then  dock  the  device  once 
they  return  to  the  AMU. 

2.  Initialization  of  the  hand  held  device  (HHD).  In  order  to  use  POMX  the  HHD 
must  re-sync  (initialize)  with  the  POMX  server.  The  initialization  process  was 
designed  to  only  take  approximately  30  seconds,  but  the  process  was  not  working. 
Some  HHD  basically  “locked  up”  during  the  process.  This  issue  is  scheduled  to 
be  fixed  with  the  release  of  version  3.0,  but  this  release  continues  to  suffer  delays. 

Workforce  Barriers:  Changing  the  mind  set  within  the  workforce  is 
always  a  barrier.  Maintainers  do  not  see  the  benefit  of  documenting  the  data 
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during  or  immediately  after  the  maintenance  action  is  performed.  Currently, 
maintainers  must  return  to  the  hanger/office  to  document  maintenance  thus 
allowing  the  technician  to  take  a  break.  During  the  initial  tests  some  users  made 
the  comment  that  they  saw  POMX  as  way  for  supervision  to  keep  the  maintainer 
on  the  flightline. 

Question  3:  What  needs  to  be  different  in  future  POMX  implementations? 

HI :  Leadership/Management:  It  is  believed  that  Leadership  will  accept 

POMX  version  3.0  once  the  software  begins  to  work  on  regular  basis.  Need  to 
thoroughly  demonstrate  a  fully  functional  capability. 

Technical:  Software  must  work  with  limited-to-no  interruptions.  If 
WLAN  connectivity  is  unavailable,  then  the  store  forward  capability  must  kick  in 
automatically.  Synchronization  with  the  POMX  server  and  the  feed  to  IMDS 
must  work  with  limited  data  rejects. 

Workforce:  Maintainers  must  change  their  mind  set.  As  Interactive 
Electronic  Technical  Manuals  (IETMs)  come  on-line  the  maintainer  will  be 
forced  to  use  the  technology.  POMX  will  become  an  integrated  part  of  the 
maintainer’ s  capability  suite. 

Based  on  the  responses  of  the  first  three  questions,  a  fourth  question  seemed  appropriate. 
Question  4:  Are  these  issues  being  addressed  in  the  current  POMX  version  3.0  implementation 
at  Hurlburt  Field? 

Network  connectivity  issues  have  been  adequately  addressed  according  to  the 
AFSOC/A4  Logistics  Integration  Superintendent.  Hurlburt  is  constrained  by  the  software 
however.  AF/ILM,  along  with  the  AIT  PMO  is  taking  a  hard  stance  regarding  the  software.  The 
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contractor  is  well  overdue  for  the  latest  version,  and  the  Air  Force  is  refusing  to  accept  the 
deliverable  until  the  problems  are  fixed.  The  frustrations  of  the  users  have  been  taken  into 
account,  and  the  Air  Force  appears  set  on  regaining  the  trust  and  confidence  of  its  Ainnen  while 
equipping  them  with  a  system  that  perfonns  as  intended. 

Cross-case  Analysis 

The  interview  responses  provided  insightful  perspectives  regarding  the  use  of  POMX  at 
different  bases.  No  two  bases  experienced  exactly  the  same  implementation  issues,  yet  there 
were  some  obvious  commonalities  and  trends  that  are  worth  further  exploration. 

Question  1:  What  efficiencies  have  been  experienced  by  using  the  POMX  or  IMIS  systems  when 
they  operate  as  they  should? 

Two  themes  emerged  as  efficiencies  gained  using  POMX.  The  first  was  that  POMX 
saves  travel  time  to  and  from  the  AMU.  Two  respondents  from  Langley,  four  from  Nellis,  and 
one  from  Randolph  provided  this  answer,  for  a  total  of  58%  of  all  respondents.  The  second 
theme  was  that  POMX  seemed  to  improve  data  accuracy  by  avoiding  the  dilution  of  data  that 
occurs  with  the  passage  of  time.  Two  respondents  from  Nellis  and  one  from  Randolph  felt  that 
capturing  the  data  sooner  increases  the  accuracy  and  detail  level  of  the  data.  This  accounted  for 
25%  of  the  respondents. 

Question  2:  What  were  the  implementation  barriers?  (What  slowed  you  down  or  kept  you  from 
using  the  system?  What  problems  existed/still  exist?) 


Table  16.  Implementation] 

3arriers 

Mgmt/Ldrshp 

Technical 

Workforce 

Lack  of  good 
implementation  plan 
(25%) 

Network 

Connectivity/Coverage 

(75%) 

Frustration  with  system  due  to 
poor  network  reliability  (33%) 

Contractor  rep  not  in 
maintenance  unit  (25%) 

Lack  of  training  (33%) 

54 


A  number  of  common  barriers  were  cited  by  respondents  across  the  different  bases.  The 
more  frequent  responses  to  this  question  are  shown  in  Table  16.  The  lack  of  a  good 
implementation  plan  was  cited  by  one  respondent  from  Langley,  one  from  Nellis,  and  one  from 
Randolph.  The  network  connectivity  barrier  was  cited  by  9  respondents  in  all,  two  from 
Langley,  4  from  Nellis,  2  from  Randolph,  and  one  from  Hurlburt.  Frustration  faced  by 
maintainers  due  to  these  network  problems  was  cited  by  one  respondent  from  Langley,  two  from 
Nellis,  and  one  from  Randolph.  The  lack  of  a  contractor  representative  in  the  maintenance  unit 
to  assist  maintainers  with  system  problems  came  from  one  individual  at  Langley,  one  at  Nellis, 
and  one  at  Randolph.  Finally,  lack  of  adequate  training  was  cited  by  one  from  Langley,  two 
from  Nellis,  and  one  from  Randolph. 

Question  3:  What  needs  to  be  different  in  future  POMX  implementations? 


Table  17.  Keys  to  Implementation 


Mgmt/Ldrshp 

Technical 

Workforce 

Emphasize  the 
importance  of  using  the 
system  (33%) 

Comprehensive  and 
reliable  network  coverage 
from  the  start.  (67%) 

Provide  proper  training  prior  to 
expecting  workers  to  use  the 
system  (25%) 

Three  common  areas  were  seen  as  important  to  POMX  implementation  as  shown  in 
Table  17.  Two  respondents  from  Nellis  and  two  from  Randolph  felt  that  management  and 
leadership  needed  to  do  a  better  job  of  stressing  the  importance  of  maintainers  using  the  system. 
Ensuring  comprehensive  and  reliable  network  coverage  from  the  beginning  of  the 
implementation  effort  was  cited  by  one  respondent  from  Langley,  four  from  Nellis,  two  from 
Randolph,  and  one  from  Hurlburt.  Finally,  the  necessity  of  providing  proper  training  to  workers 
before  expecting  them  to  use  the  system  was  cited  by  one  from  Langley  and  two  from  Randolph. 
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Summary 

This  chapter  reported  the  results  of  the  tests  of  hypotheses  as  well  as  responses  from 
interviews  conducted  at  several  bases  with  POMX  systems.  Each  test  of  hypothesis  provided  a 
small  but  necessary  piece  of  information  used  to  build  the  analytical  foundation  of  this  research. 
The  interview  responses  were  used  to  construct  a  cross-case  analysis  identifying  some  common 
trends  regarding  efficiencies,  implementation  barriers,  and  keys  to  successfully  implementing 
POMX.  The  most  common  efficiency  was  that  POMX  saves  travel  time,  the  most  common 
implementation  barrier  was  that  POMX  suffered  from  network  connectivity  and  coverage 
problems,  and  the  number  one  key  to  successful  implementation  was  to  ensure  that  the  network 
has  good  coverage  from  the  beginning  of  the  implementation.  The  next  chapter  makes 
conclusions  regarding  this  research,  offers  suggestions  and  opinions  on  the  current  state  of 
POMX  development,  and  highlights  areas  for  future  research  that  will  contribute  to  the  body  of 
knowledge  regarding  POMX  and  MDC  error  rates. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  summarizes  the  research  effort  and  answers  the  research  question. 
Observations  made  based  on  the  results  of  the  analysis  are  presented,  along  with 
recommendations  for  future  research  related  to  this  study.  The  research  question  is  restated, 
followed  by  a  discussion  of  the  investigative  questions,  research  conclusions,  and 
recommendations  for  future  research. 

Research  Question 

The  overarching  question  for  this  research  is:  Does  implementation  of  Point-of- 
Maintenance  (POMX)  at  Randolph  AFB,  Texas  provide  significant  improvement  to  flightline 
maintenance  data  accuracy  over  the  legacy  method  of  inputting  data  via  IMDS  computer 
tenninals  located  at  the  aircraft  maintenance  unit  (AMU)?  Based  on  the  data  provided  by 
Randolph  AFB,  there  was  no  significant  evidence  to  suggest  that  current  POMX 
implementations  provide  any  improved  accuracy  benefits  to  flightline  maintenance  data  over 
legacy  MDC  methods.  This  conclusion  is  further  substantiated  by  the  answers  to  the 
investigative  questions. 

Investigative  Question  1 

Does  the  POMX  system  at  Randolph  AFB,  Texas  provide  a  significant  reduction  in  MDC  errors 
when  compared  to  legacy  systems? 

The  test  results  from  hypothesis  1  indicate  that  there  is  not  enough  statistical  evidence  at 
the  .05  level  of  significance  to  reject  the  null  hypothesis  that  the  POMX  mean  error  rate  at 
Randolph  AFB  is  equal  to  or  greater  than  the  NON  POMX  mean  error  rate.  In  other  words,  there 
is  no  support  for  the  research  hypothesis  that  POMX  mean  error  rates  are  less  than  NON  POMX 
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error  rates.  The  position  of  this  research  is  that  the  POMX  system  at  Randolph  AFB  does  not 
currently  provide  a  significant  reduction  in  MDC  error  rates  when  compared  to  legacy  systems. 

Investigative  Question  2 

Are  specific  types  of  MDC  errors  at  Randolph  AFB,  TX  reduced  by  using  POMX  when  compared 
to  legacy  systems? 

This  analysis  did  not  provide  any  statistically  significant  results  at  the  .05  significance 
level  to  reject  the  null  hypothesis  that  the  mean  number  of  different  types  of  POMX 
discrepancies  is  equal  to  or  greater  than  the  mean  number  of  NON  POMX  discrepancies  at 
Randolph  AFB.  In  other  words,  there  is  no  support  for  the  research  hypothesis  that  the  POMX 
mean  number  of  errors  is  less  than  the  NON  POMX  mean  number  of  errors.  There  remains  the 
possibility  that  a  Type  II  error  has  occurred  and  the  null  hypothesis  was  not  rejected  when  should 
have  been,  therefore  this  study  does  not  claim  that  the  POMX  mean  is  in  fact  equal  to  or  greater 
than  the  NON  POMX  mean.  Instead,  this  study  takes  the  position  that  there  is  insufficient 
evidence  to  say  that  the  mean  number  of  discrepancies  is  less  with  POMX,  and  therefore  the 
status  quo  (null)  position  is  the  default  position. 

Post  Hoc  Analysis 

The  post  hoc  interview  results  indicated  that  maintainers  are  frustrated  with  the  POMX 
system  due  to  the  network  issues  that  they  have  faced.  Some  users  stated  that  they  felt  as  if  they 
were  being  used  to  test  a  system  that  was  not  actually  ready  to  be  implemented.  This  study  was 
unable  to  determine  precisely  how  often  the  system  is  down  due  to  connectivity  problems,  but  it 
is  believed  that  this  downtime  is  a  major  factor  in  the  lack  of  more  significant  results  because 
during  these  periods  users  revert  to  the  legacy  methods  of  data  entry.  The  interviews  also 
confirmed  that  users  and  system  administrators  are  both  aware  of  and  concerned  with  the 
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volatility  of  the  wireless  networks.  This  concern  was  voiced  by  individuals  contacted  at  multiple 
locations,  including  Nellis  AFB,  Nevada,  Randolph  AFB,  Texas,  and  Langley  AFB,  Virginia.  . 
The  demands  of  flying  and  maintenance  schedules  often  led  these  users  to  abandon  POMX  after 
repeated  attempts  to  use  the  system  resulted  in  lost  connections  and  having  to  re-input  data  once 
they  were  disconnected.  Rather  than  waste  the  time  by  rolling  the  dice  on  the  wireless 
connection,  maintainers  stated  a  preference  to  use  what  they  knew  would  work  so  that  they  could 
complete  the  documentation  and  move  on  to  their  next  job. 

An  additional  observation  of  note  involves  the  two  respondents  from  Nellis  that  felt  data 
“seemed  to  be”  more  accurate  or  less  diluted  when  using  POMX.  While  this  is  a  favorable 
observation  and  a  good  indicator  that  these  maintainers  understand  the  potential  benefits  to 
POMX,  this  was  not  taken  to  mean  that  they  had  actually  realized  these  benefits.  The 
quantitative  analysis  at  Randolph  AFB  shows  that  the  perceived  benefit  has  not  been  realized  at 
that  location. 

Research  Conclusions 

The  quantitative  analysis  did  not  indicate  that  POMX  had  any  effect  on  reducing  the 
number  of  data  entry  errors,  and  this  is  likely  the  result  of  several  factors.  Foremost  is  the  fact 
that  the  bases  explored  are  greatly  underutilizing  their  POMX  systems.  The  data  at  Randolph 
AFB  after  POMX  implementation  was  essentially  the  same  as  before  POMX  implementation. 
Reasons  for  this  underutilization  included  network  connectivity  issues,  loss  of  faith  in  the  POMX 
system  by  maintenance  personnel,  the  addition  of  bulky  equipment  to  an  already  burdened 
maintainer,  lack  of  adequate  training,  and  lack  of  communication  among  organizational  entities. 

The  interviews  highlighted  two  main  themes  important  to  successfully  implementing 
POMX:  Network  connectivity  and  user  training.  While  not  to  be  taken  in  isolation  from  the 
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remainder  of  the  responses,  these  two  issues  permeated  the  responses  as  both  implementation 
barriers  to  past  efforts  and  keys  to  successful  implementation  for  future  POMX  endeavors. 

Network  connectivity  was  cited  by  75%  of  respondents  across  all  four  bases  queried  as  a 
technical  barrier,  and  it  was  cited  by  67%  of  respondents  as  being  a  key  to  successful  future 
implementations.  Many  users  at  these  bases  felt  the  frustration  of  having  to  reaccomplish  their 
MDC  due  to  losing  a  network  connection.  It  is  not  practical  to  expect  users  to  adopt  a  system 
that  regularly  increases  their  workload  and  causes  rework  due  to  connection  problems.  It  seems 
that  both  contractors  and  leadership  need  to  do  a  better  job  with  future  implementations  to  ensure 
this  type  of  problem  does  not  occur.  In  the  future,  rigorous  site  surveys  should  continue  to  be 
emphasized  to  ensure  the  contractor  offers  the  proper  network  coverage  before  the  system  is 
deemed  ready  for  use.  While  all  of  the  bases  studied  here  have  experienced  network 
connectivity  problems  since  they  installed  POMX,  the  reports  from  the  field  are  that  these  issues 
are  slowly  resolving  themselves,  and  Hurlburt  Field  reports  good  coverage  throughout  in 
anticipation  of  the  next  round  of  user  evaluations. 

Training  was  not  perceived  to  be  as  problematic  as  network  connectivity;  still  it  was  cited 
by  33%  of  respondents  as  a  barrier  to  successful  implementation,  and  by  25%  of  respondents  as  a 
key  to  successful  implementation  in  the  future.  There  is  a  positive  outlook  for  the  future 
regarding  training,  as  the  current  POMX  effort  incorporates  both  detailed  training  modules  and 
quick  reference  sheets.  A  comprehensive  user’s  guide  will  also  function  as  both  a  training  tool 
and  a  permanent  reference.  These  documents  are  in  various  iterations  of  development,  yet  each 
has  significant  fidelity  at  this  point  in  time.  This  appears  to  be  a  very  positive  step  toward 
alleviating  the  types  of  training  concerns  stated  in  the  interview  responses. 
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Based  on  the  many  news  releases  over  the  past  three  years,  it  is  surprising  that  these 
existing  POMX  systems  at  Langley,  Nellis,  and  Randolph  are  being  so  underutilized.  Estimates 
at  the  bases  range  from  1-2%  to  50%  of  jobs  being  input  at  the  point-of-maintenance.  The  use  of 
IETMs  is  hoped  to  help  spur  the  use  of  the  wireless  technology,  and  conversion  efforts  among 
the  various  weapon  systems  should  continue  to  be  a  priority. 

While  implementation  of  POMX  systems  has  been  at  the  MAJCOM  level,  there  is  a  need 
for  a  feedback  link  to  ensure  lessons  learned  are  captured  for  the  benefit  of  the  current  POMX 
acquisition  program  at  AFMC.  Some  interviewees  were  concerned  with  the  amount  of 
communication  occurring  between  the  Communications  squadron  and  the  Aircraft  Maintenance 
squadron.  A  good  communication  link  should  also  exist  between  these  two  entities  to  ensure 
POMX  user  issues  are  fully  understood  and  adequately  addressed.  The  Communications 
squadron  maintains  the  network  and  functions  as  a  key  enabler  for  the  maintainers  using  POMX; 
therefore  they  should  understand  the  maintenance  perspective.  POMX  will  succeed  over  time, 
but  each  new  base  that  implements  POMX  would  do  well  to  study  past  implementation  efforts. 
There  are  also  opportunities  to  make  successful  implementation  more  probable  through 
additional  research. 

Recommendations  for  Future  Research 

The  POMX  version  3.0  software  user  evaluations  at  Hurlburt  Field  are  scheduled  for 
August,  2006.  This  software,  coupled  with  the  error  checking  capability  of  the  POMX  server,  is 
expected  to  provide  a  significant  reduction  in  data  collection  errors.  A  case  study  of  the  POMX 
effort  at  Hurlburt  Field  could  capture  not  only  past  successes  and  failures,  but  also  the  latest  user 
feedback  and  data  error  rates.  The  error  rates  could  then  be  analyzed  and  compared  with  the 
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results  of  this  study  to  see  if  the  Air  Force  is  getting  better  at  implementing  POMX,  or  if  they  are 
making  the  same  mistakes. 

Another  area  of  interest  falls  into  the  predictability  of  data  errors.  While  Folmar  and 
Determan  used  a  survey  to  detennine  the  potential  causes  of  MDC  errors,  an  opportunity  seems 
to  exist  for  developing  a  regression  model  to  assist  in  predicting  when  MDC  errors  are  more 
likely  to  occur.  Variables  such  as  rank,  Air  Force  Specialty  Code  (AFSC),  years  of  experience, 
career  development  course  score,  and  EPR  ratings  may  prove  useful  in  helping  identify  those 
individuals  who  are  most  likely  to  have  difficulty  entering  correct  data.  Early  identification  of 
these  individuals  can  help  prevent  inaccurate  data  entry  through  a  tailored  training  program  and 
emphasis  on  proper  supervision. 

A  global  study  based  on  the  concepts  from  the  post  hoc  analysis  of  Chapter  4  would  help 
identify  the  problems  and  issues  being  faced  by  all  bases  implementing  POMX.  Advantages  and 
disadvantages  of  different  types  of  hardware  might  result  in  a  best-of-breed  configuration  that 
bases  can  use  as  a  template  for  updating  their  infrastructure. 

Finally,  a  deployable  POMX  capability  needs  to  be  studied  to  help  identify  requirements 
such  as  equipment,  additional  personnel  (Comm.),  security  concerns,  and  a  host  of  other  factors 
that  affect  our  potential  ability  to  use  POMX  in  a  deployed  location. 

Summary 

This  chapter  reviewed  the  research  effort  and  answered  the  research  question  through  the 
use  of  the  two  investigative  questions  and  the  post  hoc  interviews.  The  conclusion  is  that  POMX 
at  Randolph  AFB  does  not  currently  provide  a  significant  increase  to  data  accuracy  over  legacy 
MDC  entry  methods.  Underutilization  of  POMX  is  an  issue  across  multiple  bases,  and  there  are 
various  factors  contributing  to  this  scenario.  Several  areas  for  future  research  were  presented, 
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including  a  global  survey  of  issues  and  problems  at  all  POMX  bases,  a  regression  model  to 
predict  which  individuals  are  more  prone  to  making  data  entry  errors,  a  case  study  based  on  the 
next  round  of  POMX  version  3.0  user  evaluations  at  Hurlburt  Field  in  August  2006,  and  an 
evaluation  of  deployable  POMX  requirements. 
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Appendix  A.  Error  Rate  Table 


Month  -  Yr. 

POMX  Status 

Jobs  in  Error  Jobs  Reviewed 

Error  Rate 

Nov-05 

POMX 

4 

20 

0.200 

Oct-05 

POMX 

5 

21 

0.238 

Sep-05 

POMX 

7 

20 

0.350 

Aug-05 

POMX 

8 

20 

0.400 

Jul-05 

POMX 

6 

20 

0.300 

Jun-05 

POMX 

7 

20 

0.350 

May-05 

POMX 

4 

21 

0.190 

Apr-05 

POMX 

3 

20 

0.150 

Mar-05 

POMX 

3 

20 

0.150 

Dec-04 

POMX 

10 

20 

0.500 

Nov-04 

POMX 

3 

20 

0.150 

Oct-04 

POMX 

6 

20 

0.300 

Sep-04 

POMX 

8 

20 

0.400 

Aug-04 

POMX 

8 

21 

0.381 

Jul-04 

POMX 

4 

20 

0.200 

Jun-04 

POMX 

2 

20 

0.100 

May-04 

POMX 

3 

20 

0.150 

Apr-04 

POMX 

9 

18 

0.500 

Mar-04 

POMX 

12 

32 

0.375 

Feb-04 

POMX 

2 

27 

0.074 

Nov-03 

POMX 

6 

20 

0.300 

Oct-03 

POMX 

3 

22 

0.136 

Aug-03 

POMX 

3 

20 

0.150 

May-03 

NON  POMX 

3 

24 

0.125 

Apr-03 

NON  POMX 

1 

21 

0.048 

Mar-03 

NON  POMX 

2 

20 

0.100 

Jan-03 

NON  POMX 

5 

20 

0.250 

Dec-02 

NON  POMX 

4 

23 

0.174 

Nov-02 

NON  POMX 

5 

20 

0.250 

Oct-02 

NON  POMX 

4 

20 

0.200 

Sep-02 

NON  POMX 

4 

20 

0.200 

Aug-02 

NON  POMX 

3 

20 

0.150 

Jul-02 

NON  POMX 

1 

20 

0.050 

Jun-02 

NON  POMX 

3 

20 

0.150 

May-02 

NON  POMX 

6 

22 

0.273 

Apr-02 

NON  POMX 

3 

21 

0.143 

Mar-02 

NON  POMX 

8 

13 

0.615 

Feb-02 

NON  POMX 

6 

12 

0.500 

Jan-02 

NON  POMX 

3 

11 

0.273 

Nov-01 

NON  POMX 

2 

28 

0.071 

Jun-01 

NON  POMX 

2 

20 

0.100 

Mar-01 

NON  POMX 

3 

20 

0.150 

Jan-01 

NON  POMX 

4 

20 

0.200 

Dec-00 

NON  POMX 

18 

29 

0.621 
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Oct-OO 

NON  POMX 

3 

10 

0.300 

Sep-00 

NON  POMX 

6 

38 

0.158 

Jul-00 

NON  POMX 

2 

22 

0.091 

Jun-00 

NON  POMX 

8 

21 

0.381 

May-00 

NON  POMX 

12 

28 

0.429 

Apr-00 

NON  POMX 

6 

25 

0.240 

Mar-00 

NON  POMX 

12 

44 

0.273 

Feb-00 

NON  POMX 

10 

41 

0.244 

Jan-00 

NON  POMX 

11 

37 

0.297 

Dec-99 

NON  POMX 

9 

29 

0.310 

Nov-99 

NON  POMX 

17 

51 

0.333 

Oct-99 

NON  POMX 

16 

35 

0.457 

Sep-99 

NON  POMX 

8 

29 

0.276 

Aug-99 

NON  POMX 

23 

68 

0.338 

Jul-99 

NON  POMX 

17 

43 

0.395 

Jun-99 

NON  POMX 

21 

25 

0.840 

May-99 

NON  POMX 

21 

42 

0.500 

Mar-99 

NON  POMX 

35 

58 

0.603 

Feb-99 

NON  POMX 

40 

74 

0.541 
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Appendix  B:  Number  of  Discrepancies  Table 


Month  -  Yr 

POMX  Status 

WCE NARR 

AT 

wuc 

HMAL 

WD 

Nov-05 

POMX 

1 

0 

2 

3 

0 

Oct-05 

POMX 

0 

1 

3 

3 

2 

Sep-05 

POMX 

0 

3 

1 

4 

0 

Aug-05 

POMX 

1 

2 

3 

5 

0 

Jul-05 

POMX 

0 

2 

0 

6 

0 

Jun-05 

POMX 

0 

1 

3 

5 

0 

May-05 

POMX 

0 

0 

2 

2 

0 

Apr-05 

POMX 

0 

1 

2 

0 

1 

Mar-05 

POMX 

0 

1 

3 

3 

0 

Dec-04 

POMX 

0 

5 

5 

9 

0 

Nov-04 

POMX 

1 

1 

1 

2 

0 

Oct-04 

POMX 

0 

1 

3 

5 

0 

Sep-04 

POMX 

0 

2 

2 

7 

0 

Aug-04 

POMX 

1 

3 

3 

3 

0 

Jul-04 

POMX 

0 

0 

1 

4 

0 

Jun-04 

POMX 

0 

2 

0 

0 

0 

May-04 

POMX 

0 

0 

0 

3 

0 

Apr-04 

POMX 

2 

2 

3 

5 

0 

Mar-04 

POMX 

0 

1 

1 

12 

0 

Feb-04 

POMX 

0 

0 

1 

2 

0 

Nov-03 

POMX 

1 

1 

5 

2 

0 

Oct-03 

POMX 

0 

1 

1 

2 

0 

Sep-03 

POMX 

0 

0 

0 

0 

0 

Aug-03 

POMX 

0 

1 

2 

1 

0 

May-03 

NON  POMX 

0 

1 

1 

1 

0 

Apr-03 

NON  POMX 

0 

1 

0 

1 

0 

Mar-03 

NON  POMX 

0 

0 

2 

0 

0 

Jan-03 

NON  POMX 

0 

2 

4 

1 

0 

Dec-02 

NON  POMX 

0 

0 

1 

3 

0 

Nov-02 

NON  POMX 

0 

0 

4 

1 

0 

Oct-02 

NON  POMX 

0 

0 

3 

1 

0 

Sep-02 

NON  POMX 

0 

0 

3 

1 

0 

Aug-02 

NON  POMX 

0 

0 

3 

0 

0 

Jul-02 

NON  POMX 

0 

1 

0 

0 

0 

Jun-02 

NON  POMX 

0 

0 

0 

3 

0 

May-02 

NON  POMX 

0 

4 

3 

2 

0 

Apr-02 

NON  POMX 

1 

0 

1 

2 

0 

Mar-02 

NON  POMX 

0 

0 

8 

0 

0 

Feb-02 

NON  POMX 

0 

3 

2 

1 

0 

Jan-02 

NON  POMX 

0 

0 

3 

1 

0 

Nov-01 

NON  POMX 

0 

1 

0 

1 

1 

Jun-01 

NON  POMX 

0 

1 

0 

1 

1 

Mar-01 

NON  POMX 

0 

0 

2 

0 

0 

Jan-01 

NON  POMX 

3 

0 

1 

0 

0 

Dec-00 

NON  POMX 

15 

0 

3 

0 

0 

Oct-OO 

NON  POMX 

7 

3 

1 

2 

0 

Sep-00 

NON  POMX 

1 

3 

3 

0 

0 

Jul-00 

NON  POMX 

2 

1 

3 

1 

1 
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Appendix  C:  Tests  of  t-statistic  Assumptions 


Hypothesis  1 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


Distributions  POMX  Status=NON  POMX 


Moments 

m m 

Fitted  Normal 

— ■■ ■ 

Mean 

0.2163333 

Parameter  Estimates 

1 

Std  Dev 

Std  Err  Mean 

upper  95%  Mean 

0.1576542 

0.032181 

0.2829049 

0.1497618 

Type  Parameter  Estimate 

Location  p  0.2163333 

Dispersion  a  0.1576542 

Low  er  95% 

0.1497618 

0.122531 

Upper  95% 

0.2829049 

0.2211511 

N 

24 

Goodness-of-Fit  Test 

1 

Shapiro- Wilk  W  Test 


W 

0.807421 


Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 


- Normal(0.21 633,0. 15765) 


Ho. 


Note  the  outliers  shown  in  the  box-plot  above  for  the  NON  POMX  data  set  of  overall  Error  Rate. 
This  results  in  the  p-value  of  .0004  <  .05.  The  data  are  not  approximately  nonnal 


Distributions  POMX  Status=NON  POMX 


Error  Rate 


Moments 

_ 1 

Fitted  Normal 

Mean 

0.1798182 

Parameter  Estimates 

i 

Std  Dev 

Std  Err  Mean 

upper  95%  Mean 

0.1022689 

0.0218038 

0.2251616 

0 1344747 

Type  Parameter  Estimate 

Location  p  0.1798182 

Dispersion  a  0.1022689 

Low  er  95% 

0.1344747 

0.0786807 

Upper  95% 

0.2251616 

0.1461488 

N 

22 

Goodness-of-Fit  Test 

_ ) 

-Normal(0. 17982, 0.1 0227) 


Shapiro- Wilk  W  Test 

W  Pcoh^w 

0.893231  (  j 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


The  test  is  repeated  with  the  outliers  removed,  which  results  in  a  p-value  of  .0218  <  .05.  Though 
still  not  quite  normal,  the  population  is  assumed  to  be  normal  for  testing  purposes. 


Distributions  POMX  Status=POMX 


The  POMX  data  set  are  normally  distributed. 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  equal. 


Level 

Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

0.1576542 

0.1126111 

0.1050000 

POMX 

24 

0.1347980 

0.1154861 

0.1143333 

Test 

F  Ratio 

DFNum 

DFDen 

p-Value 

0'Brien[.5] 

0.3861 

1 

46 

0.5374 

Brown-Forsythe 

0.0966 

1 

46 

0,7^74 

Levene 

0.0125 

1 

46 

c~z 

Bartlett 

0.5500 

1 

0?SS3 

F  Test  2-sided 

1.3679 

23 

23 

0.4584 

Welch  Anova  testing  Means  Equal,  allow  ing  Std  Devs  Not  Equal 
F  Ftetio  DFNum  DFDen  FTob  >  F 
0.7030  1  44.916  0.4062 


t  Test 
0.8384 
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Hypothesis  2 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


AT 


Moments 

■Ml 

Fitted  Normal 

Mean 

0.875 

Parameter  Estimates 

1 

Std  Dev 

Std  Err  Mean 

upper  95%  Mean 

1.2269616 

0.2504525 

1.3931004 

0  3568996 

Type  Parameter  Estimate 

Location  p  0.875 

Dispersion  a  1.2269616 

Low  er  95% 

0.3568996 

0.9536112 

Upper  95% 

1.3931004 

1.7211334 

N 

24 

Goodness-of-Fit  Test 

■■1 

-  Normal(0.875, 1.22696) 


Shapiro- Wilk  W  Test 
W 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


0.732481 


The  NON  POMX  data  are  not  normally  distributed. 


Distributions  POMX  Status=POMX 
AT 


- Normalf  1.291 67, 1.1 9707) 


Moments 

mm 

Fitted  Normal 

Mean 

1.2916667 

Parameter  Estimates 

MMI 

Std  Dev 

Std  Err  Mean 

upper  95%  Mean 

1.1970677 

0.2443504 

1.797144 

0  7861893 

Type  Parameter  Estimate 

Location  p  1.2916667 

Dispersion  a  1.1970677 

Low  er  95% 

0.7861893 

0.9303773 

Upper  95% 

1.797144 

1.6791994 

N 

24 

Goodness-of-Fit  Test 

Shapiro- Wilk  W  Test 

W  Pr~u-fi/v 

0.838833  C  J 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


The  POMX  data  are  not  normally  distributed. 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  equal. 


Level  Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

1 .226962 

0.9479167 

0.8750000 

POMX 

24 

1.197068 

0.8888889 

0.7916667 

Test 

F  Ratio  DFNum 

DFDen 

p-Value 

0'Brien[.5] 

0.0086 

1 

46 

0.9264 

Brow  n-Forsythe 

0.0702 

1 

46 

Levene 

0.0711 

1 

46  ( 

Bartlett 

0.0137 

1 

0/ 

°°\ 

F  Test  2-sided 

1 .0506 

23 

23 

0.9069 

Welch  Anova  testing  Means  Equal,  allow  ing  Std  Devs  Not  Equal 
F  Ratio  DFNum  DFDen  Prob  >  F 
1.4180  1  45.972  0.2398 

t  Test 
1.1908 
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Hypothesis  3 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


Distributions  POMX  Status=NON  POMX 


WUC 


Moments 


Mean 
Std  Dev 
Std  Err  Mean 
upper  95%  Mean 
low  er  95%  Mean 
N 


Fitted  Normal 


2.125 

1.8252457 

0.3725767 

2.8957336 

1.3542664 

24 


Parameter  Estimates 


Type  Parameter 
Location  p 
Dispersion  a 


Estimate 
2.125 
1 .8252457 


Low  er  95% 
1 .3542664 
1.4186057 


Upper  95% 
2.8957336 
2.5603827 


Goodness-of-Fit  Test 


- Normal(2. 125, 1.82525) 

The  NON  POMX  data  are  not  normally  distributed. 


Shapiro- Wilk  W  Test 

„  0 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 

Ho. 


0.852109 


Distributions  POMX  Status=POMX 


WUC 


Moments 

Fitted  Normal 

Mean  1.9583333 

Std  Dev  1.428869 

Std  Err  Mean  0.2916667 

upper  95%  Mean  2.5616918 
low  er  95%  Mean  1 .3549749 

N  24 

Parameter  Estimates 

Type  Parameter  Estimate  Low  er  95%  Upper  95% 

Location  p  1.9583333  1.3549749  2.5616918 

Dispersion  o  1.428869  1.1105364  2.0043612 

Goodness-of-Fit  Test 

-  Normal)  1.95833, 1.42887) 


Shapiro- Wilk  W  Test 
W 

0.907838  C  y 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


The  POMX  data  are  not  normally  distributed 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  equal. 


Level 

Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

1 .825246 

1.385417 

1.375000 

POMX 

24 

1 .428869 

1.131944 

1.125000 

Test 

F  Ratio 

DFNum 

DFDen 

p-Value 

0'Brien[.5] 

0.6516 

1 

46 

0.4237 

Brow  n-Forsythe 

0.7150 

1 

46 

0.4022 

Levene 

0.7583 

1 

46  ( 

<  N 

Bartlett 

1.3361 

1 

CT2477 

F  Test  2-sided 

1.6318 

23 

23 

0.2478 

Welch  Anova  testing  Means  Equal,  allow  ing  Std  Devs  Not  Equal 
F  Ratio  DFNum  DFDen  Prob  >  F 
0.1241  1  43.494  0.7264 

t  Test 
0.3522 
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Hypothesis  4 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


-  Normal(0.95833, 0.9079) 


Ho. 


The  NON  POMX  data  are  not  normally  distributed. 


Distributions  POMX  Status=POMX 


HMAL 


^<>1 —  • 

1 

-  Normal(3.66667, 2.83866) 


Moments 


Mean 
Std  Dev 
Std  Err  Mean 
upper  95%  Mean 
low  er  95%  Mean 
N 


Fitted  Normal 


3.6666667 

2.8386566 

0.5794383 

4.8653262 

2.4680071 

24 


Parameter  Estimates 


Type  Parameter 
Location  p 
Dispersion  a 


Estimate 

3.6666667 

2.8386566 


Low  er  95% 
2.4680071 
2.2062424 


Upper  95% 
4.8653262 
3.9819556 


Goodness-of-Fit  Test 


Shapiro- Wilk  W  Test 

w  O 

Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


0.900479 


The  POMX  data  are  not  normally  distributed. 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  not  equal. 


Level 

Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

0.907896 

0.638889 

0.625000 

POMX 

24 

2.838657 

2.111111 

2.000000 

Test 

F  Ratio 

DFNum 

DFDen 

p-Value 

0'Brien[.5] 

5.0434 

1 

46 

0.0296* 

Brown-Forsythe 

9.5200 

1 

46 

0.0034* 

Levene 

13.6684 

1 

46  | 

Bartlett 

24.5007 

1 

<!UUTfr 

F  Test  2-sided 

9.7758 

23 

23 

<.0001* 

Welch  Anova  testing  Means  Equal,  allowing  Std  Devs  Not  Equal 
F  Ratio  DFNum  DFDen  Prob  >  F 
19.8195  1  27.657  0.0001* 

t  Test 
4.4519 
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Hypothesis  5 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


Distributions  POMX  Status=NON  POMX 


-Normal(0. 125, 0.33783) 


The  NON  POMX  data  are  not  normally  distributed. 


Distributions  POMX  Status=POMX 


WD 


« 

IO  ■ 

h - 

-0.5 

i  i  i  i  r 

.5  1  1.5  2  2.5 

-Normal(0. 125, 0.44843) 


Moments 


Mean 
Std  Dev 
Std  Err  Mean 
upper  95%  Mean 
low  er  95%  Mean 
N 


Fitted  Normal 


0.125 

0.4484272 

0.0915348 

0.3143542 

-0.064354 

24 


Parameter  Estimates 


Type  Parameter 
Location  p 
Dispersion  a 


Estimate 

0.125 

0.4484272 


Low  er  95% 
-0.064354 
0.3485237 


Upper  95% 
0.3143542 
0.629036 


Goodness-of-Fit  Test 


Shapiro- Wilk  W  Test 
W 

0.315543 


O 


Note:  Ho  =  The  data  is  from  the  Normal  distribution.  Small  p-values  reject 
Ho. 


The  POMX  data  are  not  normally  distributed. 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  equal. 


Level  Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

0.3378320 

0.2187500 

0.1250000 

POMX 

24 

0.4484272 

0.2291667 

0.1250000 

Test 

F  Ratio  DFNum 

DFDen 

p-Value 

0'Brien[.5] 

0.2707 

1 

46 

0.6053 

Brow  n-Forsythe 

0.0000 

1 

46 

1 .0000 

Levene 

0.0124 

1 

46  ( 

> 

Bartlett 

1.7817 

1 

>S0'T5T9 

F  Test  2-sided 

1.7619 

23 

23 

0.1820 

Welch  Anova  testing  Means  Equal,  allow  ing  Std  Devs  Not  Equal 
F  Ratio  DFNum  DFDen  Prob  >  F 
0.0000  1  42.747  1.0000 


t  Test 
0.0000 
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Hypothesis  6 

Shaprio-Wilk  goodness-of-fit  test  for  a  normal  distribution  at  the  .05  level  of  significance. 


Distributions  POMX  Status=NON  POMX 


-Normal(  1.20833, 3.3229) 


The  NON  POMX  data  are  not  normally  distributed. 


Distributions  POMX  Status=POMX 


-  Normal(0.291 67, 0.55003) 


The  POMX  data  are  not  normally  distributed. 
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Levene  test  of  equal  variances  at  the  .05  level  of  significance.  The  variances  are  not  equal. 


Level 

Count 

Std  Dev 

MeanAbsDif  to  Mean 

MeanAbsDif  to  Median 

NON POMX 

24 

3.322900 

1 .847222 

1.208333 

POMX 

24 

0.550033 

0.437500 

0.291667 

Test 

F  Ratio 

DFNum 

DFDen 

p-Value 

0'Brien[.5] 

1.6120 

1 

46 

0.2106 

Brow  n-Forsythe 

1 .7777 

1 

46 

0.1890 

Levene 

6.2891 

1 

46  I 

o 

Bartlett 

50.9865 

1 

<mjr 

F  Test  2-sided 

36.4970 

23 

23 

<0001* 

Welch  Anova  testing  Means  Equal,  allow  ing  Std  Devs  Not  Equal 
F  Ratio  DFNum  DFDen  Prob  >  F 
1.7777  1  24.259  0.1948 

t  Test 
1.3333 
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